Sign in with
Sign up | Sign in
Your question
Solved

Need advice on CPU - unusual problem - What would your do ?

Last response: in CPUs
Share
May 19, 2012 4:39:53 PM

I have some fairly large data analysis problems I would like to solve. I use the multiprocessing module in Python which works well but I am coding some problems now that will have data sets of perhaps a billion items ( perhaps 50M items iterated 20 times)

I have an AMD 630 4 core CPU and for the problems I am doing now it is OK - Python can keep all cores running at near 100% for the duration of the problem but I would like to add some cheap cores. I am thinking of a AMD FX 8120 - 8 cores. At my local MicroCenter I can get the CPU and a cheap motherboard for about $210. For about $250 i can upgrade to a FX 8150. For the way I want to use the machine - keeping all cores busy all of the time - I think an 8150 would give me about the same performance as an Intel 2700 although I realize the 2700 is better at all most everything else. I have also thought about an i7 3930 with 6 cores and hyperthreading = 12 cores. The Intel solution is a LOT more expensive - roughly $800. I would still buy the Intel solution if it gave me a LOT more performance.

One more idea I have had is to just build a whole new system with an ivy bridge 3770 - 8 cores for about $400 and more than enough graphics power for my needs.

I am willing to wait until fall if something better is coming. By the way, this is just a hobby and I don't play games with my PC.

So... What would you do?
May 19, 2012 5:37:47 PM

I have some fairly large data analysis problems I would like to solve. I use the multiprocessing module in Python which works well but I am coding some problems now that will have data sets of perhaps a billion items ( perhaps 50M items iterated 20 times)

I have an AMD 630 4 core CPU and for the problems I am doing now it is OK - Python can keep all cores running at near 100% for the duration of the problem but I would like to add some cheap cores. I am thinking of a AMD FX 8120 - 8 cores. At my local MicroCenter I can get the CPU and a cheap motherboard for about $210. For about $250 i can upgrade to a FX 8150. For the way I want to use the machine - keeping all cores busy all of the time - I think an 8150 would give me about the same performance as an Intel 2700 although I realize the 2700 is better at all most everything else. I have also thought about an i7 3930 with 6 cores and hyperthreading = 12 cores. The Intel solution is a LOT more expensive - roughly $800. I would still buy the Intel solution if it gave me a LOT more performance.

One more idea I have had is to just build a whole new system with an ivy bridge 3770 - 8 cores for about $400 and more than enough graphics power for my needs.

I am willing to wait until fall if something better is coming. By the way, this is just a hobby and I don't play games with my PC.

So... What would you do?
m
0
l
May 19, 2012 5:49:53 PM

for encoding the amd bulldozer cpus are about equal or a little worse than the intel cpu you listed.
m
0
l
Related resources
a c 103 à CPUs
May 19, 2012 6:07:34 PM

Go with a FX-8120 or two, core v cost for what you want is a no brainer.

I take it raw cpu speed is not as important as more cores.

Don't use hyperthreading, real cores better than virtuals.
m
0
l
a b à CPUs
May 19, 2012 6:10:08 PM

Hi :) 

Multiple posting is against the rules here...reported...

All the best Brett :) 
m
0
l
a c 79 à CPUs
May 19, 2012 6:33:11 PM

for a fraction of that price could you not rent some resource from amazon's cloud?

Unless you want to update?

http://aws.amazon.com/ec2/

I'm sure there are other could based resources.
m
0
l
May 19, 2012 7:11:51 PM

13thmonkey said:
for a fraction of that price could you not rent some resource from amazon's cloud?

Unless you want to update?

http://aws.amazon.com/ec2/

I'm sure there are other could based resources.


(Sound of person slapping forehead) Great idea! i don't know if it would work or not or how steep the learning curve would be. I see Amazon lets you try a micro instance for free so perhaps I could get some idea of what I need from that. Otherwise , for a few $10 of dollars it would be worth running a few trials. Thanks for the idea.
m
0
l
May 19, 2012 7:16:18 PM

amdcpus said:
for encoding the amd bulldozer cpus are about equal or a little worse than the intel cpu you listed.


I think you meant the 2700 but I also mentioned the 3930 and the 3770. Did you mean one of those? And compared to the AMD 8120 right?
m
0
l
a c 79 à CPUs
May 19, 2012 7:21:45 PM

nile said:
(Sound of person slapping forehead) Great idea! i don't know if it would work or not or how steep the learning curve would be. I see Amazon lets you try a micro instance for free so perhaps I could get some idea of what I need from that. Otherwise , for a few $10 of dollars it would be worth running a few trials. Thanks for the idea.


You can load up VM's onto it, so p2V your machine (or create a virtual that just does this job only), and run that as a VM so it should just work, they also have shedloads (25 pages worth) of appliances that you can use.

The on-demand service is definitely the better option, as there is no initial cost.
m
0
l
May 19, 2012 10:01:47 PM

I notice they don't support VirtualBox. This isn't a big deal - I don't think. I could just use SUSE or Red Hat I think and import my code.

Also I couldn't find the difference between cores and compute units? For example, it looks as if I can get 20 compute units (8 virual cores) for only 66 units per hour. Would that be about the same as 20 cores in a pc?

i guess I need to run some of my old code to get a feel for this.
m
0
l

Best solution

a c 79 à CPUs
May 19, 2012 10:49:22 PM

Their compute unit explained
http://aws.amazon.com/ec2/faqs/#What_is_an_EC2_Compute_...

and i'd consider one of these
Cluster Compute Instances

Instances of this family provide proportionally high CPU resources with increased network performance and are well suited for High Performance Compute (HPC) applications and other demanding network-bound applications. You can learn more about Cluster instance concepts by reading the Amazon EC2 documentation. For more information about specific use cases and cluster management options for HPC, please visit the HPC solutions page.

Cluster Compute Quadruple Extra Large 23 GB memory, 33.5 EC2 Compute Units, 1690 GB of local instance storage, 64-bit platform, 10 Gigabit Ethernet
Cluster Compute Eight Extra Large 60.5 GB memory, 88 EC2 Compute Units, 3370 GB of local instance storage, 64-bit platform, 10 Gigabit Ethernet
at $2.40 per hour (unless I am mis reading it)
so that'd be 33.5 or 88 EC2 units worth of compute power, not sure how they are arranged as cores, as others list for instance:
High-Memory Quadruple Extra Large Instance 68.4 GB of memory, 26 EC2 Compute Units (8 virtual cores with 3.25 EC2 Compute Units each), 1690 GB of local instance storage, 64-bit platform

Thats still a lot of power for not a lot of cash.
Share
May 19, 2012 11:45:20 PM

13thmonkey said:

Thats still a lot of power for not a lot of cash.


Yeah, you're right. A 1 Gz 2007 Opteron or Xeon CPU would be just fine for my purpose. For my use, cores are more important than Gz. I also read it as $2.40 per hour.

You've sold me - I am going to set up a micro instance (free - I think) figure out what I am doing and then upload some of my old code to one of the Cluster Compute instances and see what happens. it seems almost unbelievable to me that a person could get that much compute capacity for so little.

Another advantage for me that I didn't mention before is that I travel a lot all over the world. EC2 allows me to work from anywhere where as a dedicated box only allows me to work from home.

Thanks so much for your help. I appreciate it. I'll sign off now and won't bug you any more. I'm off to climb the EC2 learning curve!
m
0
l
a c 314 à CPUs
May 20, 2012 12:08:00 AM

The differing architectures come into play.

Intel i7, like the 2700K or the ivy bridge i7-3770K actually have only 4 cores, but has 8 threads. The extra 4 threads are called hyperthreads which use the residual cycles of the main cores to do some extra useful work.
I estimate that they are worth 1/4 to 1/3 of a full core each.
Intel chips have a better internal architecture than amd, and will produce more work per clock cycle.

FX-8150 has, in theory, 8 cores in 4 groups. Each of the pairs of cores share some resources, so they are not completely independent.

And, the Intel "K" suffic chips can be conservatively overclocked from 3.5ghz to 4.0 or 4.5, increasint their productive performance.

Since multiple cores can be used by your app, it is the number of threads times the performance of each thread that will determine how long it taks to run your job.

FX- 8150 can also be overclocked, but I am less familiar with exactly how high. Probably similar to Intel.

One way to compare them would be to compare the passmark numbers.
Better would be some benchmarks which used your specific app if you can find one.
At stock speed, the FX-8150 has a passmark number of 8236.
The Intel 3770K is at 10419.
With overclocking, you might get 20-25% more.

There are some higher performing 6 core chips, but they are in the $1000 range.
Here is the complete chart:
http://www.cpubenchmark.net/high_end_cpus.html
m
0
l
a b à CPUs
May 20, 2012 12:09:29 AM

This next topics has been merged by Pyree
  • Have big data set - need more cores
    m
    0
    l
    a c 146 à CPUs
    May 20, 2012 12:24:40 AM

    I would go with a Xeon.
    m
    0
    l
    a c 79 à CPUs
    May 20, 2012 1:23:50 PM

    let us know about you get on.
    m
    0
    l
    a c 79 à CPUs
    May 20, 2012 1:24:08 PM

    Brett928S2 said:
    Hi :) 

    Multiple posting is against the rules here...reported...

    All the best Brett :) 


    nice contribution
    m
    0
    l
    May 27, 2012 1:43:23 AM

    Best answer selected by nile.
    m
    0
    l
    !