Solved

Need advice on CPU - unusual problem - What would your do ?

I have some fairly large data analysis problems I would like to solve. I use the multiprocessing module in Python which works well but I am coding some problems now that will have data sets of perhaps a billion items ( perhaps 50M items iterated 20 times)

I have an AMD 630 4 core CPU and for the problems I am doing now it is OK - Python can keep all cores running at near 100% for the duration of the problem but I would like to add some cheap cores. I am thinking of a AMD FX 8120 - 8 cores. At my local MicroCenter I can get the CPU and a cheap motherboard for about $210. For about $250 i can upgrade to a FX 8150. For the way I want to use the machine - keeping all cores busy all of the time - I think an 8150 would give me about the same performance as an Intel 2700 although I realize the 2700 is better at all most everything else. I have also thought about an i7 3930 with 6 cores and hyperthreading = 12 cores. The Intel solution is a LOT more expensive - roughly $800. I would still buy the Intel solution if it gave me a LOT more performance.

One more idea I have had is to just build a whole new system with an ivy bridge 3770 - 8 cores for about $400 and more than enough graphics power for my needs.

I am willing to wait until fall if something better is coming. By the way, this is just a hobby and I don't play games with my PC.

So... What would you do?
17 answers Last reply Best Answer
More about need advice unusual problem
  1. I have some fairly large data analysis problems I would like to solve. I use the multiprocessing module in Python which works well but I am coding some problems now that will have data sets of perhaps a billion items ( perhaps 50M items iterated 20 times)

    I have an AMD 630 4 core CPU and for the problems I am doing now it is OK - Python can keep all cores running at near 100% for the duration of the problem but I would like to add some cheap cores. I am thinking of a AMD FX 8120 - 8 cores. At my local MicroCenter I can get the CPU and a cheap motherboard for about $210. For about $250 i can upgrade to a FX 8150. For the way I want to use the machine - keeping all cores busy all of the time - I think an 8150 would give me about the same performance as an Intel 2700 although I realize the 2700 is better at all most everything else. I have also thought about an i7 3930 with 6 cores and hyperthreading = 12 cores. The Intel solution is a LOT more expensive - roughly $800. I would still buy the Intel solution if it gave me a LOT more performance.

    One more idea I have had is to just build a whole new system with an ivy bridge 3770 - 8 cores for about $400 and more than enough graphics power for my needs.

    I am willing to wait until fall if something better is coming. By the way, this is just a hobby and I don't play games with my PC.

    So... What would you do?
  2. for encoding the amd bulldozer cpus are about equal or a little worse than the intel cpu you listed.
  3. Go with a FX-8120 or two, core v cost for what you want is a no brainer.

    I take it raw cpu speed is not as important as more cores.

    Don't use hyperthreading, real cores better than virtuals.
  4. Hi :)

    Multiple posting is against the rules here...reported...

    All the best Brett :)
  5. for a fraction of that price could you not rent some resource from amazon's cloud?

    Unless you want to update?

    http://aws.amazon.com/ec2/

    I'm sure there are other could based resources.
  6. 13thmonkey said:
    for a fraction of that price could you not rent some resource from amazon's cloud?

    Unless you want to update?

    http://aws.amazon.com/ec2/

    I'm sure there are other could based resources.


    (Sound of person slapping forehead) Great idea! i don't know if it would work or not or how steep the learning curve would be. I see Amazon lets you try a micro instance for free so perhaps I could get some idea of what I need from that. Otherwise , for a few $10 of dollars it would be worth running a few trials. Thanks for the idea.
  7. amdcpus said:
    for encoding the amd bulldozer cpus are about equal or a little worse than the intel cpu you listed.


    I think you meant the 2700 but I also mentioned the 3930 and the 3770. Did you mean one of those? And compared to the AMD 8120 right?
  8. nile said:
    (Sound of person slapping forehead) Great idea! i don't know if it would work or not or how steep the learning curve would be. I see Amazon lets you try a micro instance for free so perhaps I could get some idea of what I need from that. Otherwise , for a few $10 of dollars it would be worth running a few trials. Thanks for the idea.


    You can load up VM's onto it, so p2V your machine (or create a virtual that just does this job only), and run that as a VM so it should just work, they also have shedloads (25 pages worth) of appliances that you can use.

    The on-demand service is definitely the better option, as there is no initial cost.
  9. I notice they don't support VirtualBox. This isn't a big deal - I don't think. I could just use SUSE or Red Hat I think and import my code.

    Also I couldn't find the difference between cores and compute units? For example, it looks as if I can get 20 compute units (8 virual cores) for only 66 units per hour. Would that be about the same as 20 cores in a pc?

    i guess I need to run some of my old code to get a feel for this.
  10. Best answer
    Their compute unit explained
    http://aws.amazon.com/ec2/faqs/#What_is_an_EC2_Compute_Unit_and_why_did_you_introduce_it

    and i'd consider one of these
    Cluster Compute Instances

    Instances of this family provide proportionally high CPU resources with increased network performance and are well suited for High Performance Compute (HPC) applications and other demanding network-bound applications. You can learn more about Cluster instance concepts by reading the Amazon EC2 documentation. For more information about specific use cases and cluster management options for HPC, please visit the HPC solutions page.

    Cluster Compute Quadruple Extra Large 23 GB memory, 33.5 EC2 Compute Units, 1690 GB of local instance storage, 64-bit platform, 10 Gigabit Ethernet
    Cluster Compute Eight Extra Large 60.5 GB memory, 88 EC2 Compute Units, 3370 GB of local instance storage, 64-bit platform, 10 Gigabit Ethernet
    at $2.40 per hour (unless I am mis reading it)
    so that'd be 33.5 or 88 EC2 units worth of compute power, not sure how they are arranged as cores, as others list for instance:
    High-Memory Quadruple Extra Large Instance 68.4 GB of memory, 26 EC2 Compute Units (8 virtual cores with 3.25 EC2 Compute Units each), 1690 GB of local instance storage, 64-bit platform

    Thats still a lot of power for not a lot of cash.
  11. 13thmonkey said:

    Thats still a lot of power for not a lot of cash.


    Yeah, you're right. A 1 Gz 2007 Opteron or Xeon CPU would be just fine for my purpose. For my use, cores are more important than Gz. I also read it as $2.40 per hour.

    You've sold me - I am going to set up a micro instance (free - I think) figure out what I am doing and then upload some of my old code to one of the Cluster Compute instances and see what happens. it seems almost unbelievable to me that a person could get that much compute capacity for so little.

    Another advantage for me that I didn't mention before is that I travel a lot all over the world. EC2 allows me to work from anywhere where as a dedicated box only allows me to work from home.

    Thanks so much for your help. I appreciate it. I'll sign off now and won't bug you any more. I'm off to climb the EC2 learning curve!
  12. The differing architectures come into play.

    Intel i7, like the 2700K or the ivy bridge i7-3770K actually have only 4 cores, but has 8 threads. The extra 4 threads are called hyperthreads which use the residual cycles of the main cores to do some extra useful work.
    I estimate that they are worth 1/4 to 1/3 of a full core each.
    Intel chips have a better internal architecture than amd, and will produce more work per clock cycle.

    FX-8150 has, in theory, 8 cores in 4 groups. Each of the pairs of cores share some resources, so they are not completely independent.

    And, the Intel "K" suffic chips can be conservatively overclocked from 3.5ghz to 4.0 or 4.5, increasint their productive performance.

    Since multiple cores can be used by your app, it is the number of threads times the performance of each thread that will determine how long it taks to run your job.

    FX- 8150 can also be overclocked, but I am less familiar with exactly how high. Probably similar to Intel.

    One way to compare them would be to compare the passmark numbers.
    Better would be some benchmarks which used your specific app if you can find one.
    At stock speed, the FX-8150 has a passmark number of 8236.
    The Intel 3770K is at 10419.
    With overclocking, you might get 20-25% more.

    There are some higher performing 6 core chips, but they are in the $1000 range.
    Here is the complete chart:
    http://www.cpubenchmark.net/high_end_cpus.html
  13. This next topics has been merged by Pyree
    Have big data set - need more cores
  14. I would go with a Xeon.
  15. let us know about you get on.
  16. Brett928S2 said:
    Hi :)

    Multiple posting is against the rules here...reported...

    All the best Brett :)


    nice contribution
  17. Best answer selected by nile.
Ask a new question

Read More

CPUs Data Analysis