35

I know that exact benchmark numbers will vary between manufacturers, but in general if you only consider higher end flash drives, SD cards, and USB HDDs and your Raspberry Pi is connected to a network via an ethernet switch what is the fastest configuration from a network data transfer standpoint? I'm only looking for answers in which people have actually tried the configurations listed below and have real results, not theory.

Obviously an SD card must be used because it is required for at least part of the boot cycle, but lets assume that you have an additional data partition on each of the devices listed below which has the highest network transfer speeds on the Raspberry Pi?

  1. The SD card
  2. A USB flash drive
  3. A USB HDD

This question is non-trivial because at some point processing overhead associated with one of the storage devices coupled with ethernet communication may impact transfer rate, also the ethernet port on the Raspberry Pi is controlled by the USB bus so in general throughput should be impacted by the coordination of ethernet data transfer and USB media data transfer. I do not know if the SD card is on the same shared USB bus as the USB ports and the ethernet port but if it isn't I suppose that would be a pretty large benefit performance wise.

Edited to include network transfer as requirement for benchmark

Dan B
  • 2,897
  • 2
  • 19
  • 20
  • 1
    FYI, answerers can use `sudo hdparm -t /dev/sdx` as a useful tool for benchmarking. – Jivings Jul 24 '12 at 22:39
  • 2
    I don't have benchmarks, but it's common knowledge: ssd > hdd > flash > sd > optical media – Alexander Jul 24 '12 at 22:45
  • +1 - Good point about the SD port - is it on the same bus as the USB/ethernet? Does sharing load on SD and attached USB storage work out better? – Jon Egerton Jul 25 '12 at 08:54
  • @XAleXOwnZX sure if the busses that those devices are on can match the maximum performance of the device then that is the order but I'm betting that since the Raspberry Pi only has USB your order will be more like ssd = hdd = flash > optical and SD may reside anywhere in that list based on if it has its own bus separate from the USB ports. – Dan B Jul 27 '12 at 17:49
  • That's a good point, but HDD > Flash for sure. My USB hard drive gets 30 mb/s compared to my flash drive's 5-10 – Alexander Jul 27 '12 at 18:17
  • Note that a USB3 Ethernet dongle is faster than the built-in 100 Mbps Ethernet. – Thorbjørn Ravn Andersen Jan 13 '17 at 16:06

4 Answers4

35

In benchmarking you must always establish what your limits are. Because if you expect to get 100mbs out of that lan than you are only fooling your self!

Look at this Block design of the RaspberryPI Model-B enter image description here

So we establish a very important fact here. Ethernet is bottlenecked by the USB controller because form the block digram we establish it is connected to the USB hub. (No clear indication is it uses another bus or just simply USB2?)

Ohh look - another block diagram,

enter image description here

Now that sheds even more light on the situation. The 10/100 controller is connected to the USB hub- unfortunately, again, no where on the spec does it say how fast the hub communicates with the lan controller-- expect for the key fact that is says the usb speed and lan are negotiated for mixed speed usb environments.

Where is the SD Card?

It turns out the SD card is directly connected to the BCM2835 (page 65) and they go into great detail how performance is impacted from various configuration levels. One important thing to notice is that if the SD card has a dedicated Clock it can run independently of the core CPU/GPU at full speed.(of whatever card and standard is used - you can see it supported a few standards.)

enter image description here

So what does that mean?

It means if you benchmark the faster SD card and crappy pen drive(4gb/8gb) you will most likely get massive performance differences. So now it raises the question, how does the CPU/GPU handle communication with this wonderful embedded device (usb/lan chip) and what speeds is it capable of communicating at.

You see how far theory can actually go before we actually do any benchmarking? Another key point here is -- How does the CPU control flow of data. Does it use the separate clock channel as recommended?

IN an ideal world you would think that this LAN/USB would handle it. But that requires a MCU.. do we see an MCY anywhere in that block diagram.. NO!

So the CPU has to request IO DATA send it the USB port then it goes to the LAN port (via the Same USB HUB) Yes.. so that is going to cause some speed issues somewhere.

Also great thing to note what happens when you copy to a USB pen drive and from the SD card all via LAN.. it is going to cause some traffic.

We need to benchmark various aspects

In establishing what we are trying to benchmark we can ask a real world question.

  1. How fast does the internal USB handle data from SD?
  2. How much CPU is used to read/write to SD then to LAN?
  3. How much CPU is used to read/write to a USB Pen drive?
  4. Does copying any data directly influence the CPU usage?
  5. How well does the embedded USB/LAN chip handle reading data from 2 sources back into the LAN port and does it affect the CPU proportionally?
  6. There is no need to test if lan will get full 10mb/s because its bottlenecked by the state in which the USB hub is in.

Now that you know what you are fighting for I challenge you to answer them your self.

You might find this a very good starting point.

References

But mostly,

my own technical knowledge and assessments I have made based on my own embedded experience. The OP asked a very good question but lacks to understand that without theoretical understanding of a system you are doomed in trying to assess/solve the practical problems.

Results

(4) This answer shows some down to earth piratical testing. And it proves that doing intense data transaction directly influences the CPU (just not sure if its the SD card or the process of shifting data inside the BCM chip to the USB/LAN chip)

(6) It has been proved that the only bottle neck is going to be the source (for example a slow SD card) The USB hub manages to pump data at 90%+ but answer to questions 4 and 5 can directly influence this performance.

So doing some research and contribution of other users we are starting to establish and get preliminary results.

Here is a nice chart to help visualise what we are dealing with.

enter image description here

Community
  • 1
Piotr Kula
  • 17,168
  • 6
  • 63
  • 103
  • 1
    +1 Great answer. Though can you please provide sources for your images? – Jivings Jul 31 '12 at 18:21
  • 1
    This answer does an excellent job of adding details to the question explaining why each of the different storage media could have very different performance numbers. Thank you for the additional information that I should have added to the question myself in the first place. – Dan B Aug 01 '12 at 16:46
  • 1
    USB is still around 30 MB/s, i.e. plenty to spare for a 100 Mbit network. – Thorbjørn Ravn Andersen Aug 02 '12 at 00:14
  • @ThorbjørnRavnAndersen yes that is true. But one of my questions relates to the performance of the embedded usb hub. Haw fast is it really? and does it depend on the ARM CPU- also what happens when you copy form 2 sources via LAN.. That is something worth benchmarking. I am not saying it will not do it- but certainly could be a problem in some situations. – Piotr Kula Aug 02 '12 at 08:23
  • Sounds like you have experience in the field. Could you do actual measurements? – Thorbjørn Ravn Andersen Aug 02 '12 at 08:26
  • Yes I could but I do not have time :( I wish I could do so many things.. Can you make some clones of me maybe? – Piotr Kula Aug 02 '12 at 08:38
  • See my answer http://raspberrypi.stackexchange.com/questions/302/can-i-stream-1080p-video-from-the-pi/303#303 You can get 100 Mbps of TCP (!). –  Aug 03 '12 at 09:36
  • @ppumkin Your link to BCM2835 (page 65) appears dead :( – Slaviks Aug 29 '13 at 18:45
  • Updated the link - They moved it to the Pi server. I suppose Broadcom was getting too many visits :) I also added an excerpt of the schematic. – Piotr Kula Aug 30 '13 at 08:34
3

Obviously, the highest performance would be with a ramdisk.

That won't be much use for a network-attached-storage device, but then, the pi is not necessarily the best choice for that fixed application.

For some things where the pi's flexibility is uniquely suited, actual storage requirements may be small enough to run explicitly from a ramdisk, or perhaps implicitly from ram through the disk caching mechanism (linux ramdisks are just disk caches without a backing disk anyway).

You would still need a card to boot from, but as a side benefit of running from ram you could keep the card read-only and minimize the risk of corrupting it.

Chris Stratton
  • 993
  • 5
  • 11
  • You do not need a card to boot from. The operating system / boot loader requires it! You can boot from the Serial interface or LAN if you changed the bootloader to do so. If you were technical enough you could add 3GB ram to the RPI - but that is compeltly off topic. The fact is everything goes through a stupid USB hub! bleeeh – Piotr Kula Jul 31 '12 at 18:31
  • Isn't the bootloader, at least the one you could change, located on the SD card? You might be able to get an internal boot rom to load from another source if you change some configuration pin strappings though. – Chris Stratton Jul 31 '12 at 19:03
  • I am almost positive the BCM chip.. the same like arduino for example has a bootloader.. (like BIOS on PC) There has to be some initial programme to load the next programme and so on. changing the bootlaoder on RPi.. wait a year or two. – Piotr Kula Jul 31 '12 at 19:08
  • An arduino's ATMEGA is an example of a microcontroller designed to run stand alone, with internal flash. A booloader there is user customizable. The pi's SOC on the other hand seems to be from the class of larger system parts which require external memories. These typically do have a boot loader, but it's a mask rom that cannot be altered once they leave the factory. In a traditional product based on something like that, there would be a 4-8 MByte external flash chip. But the pi seems to use the removable SD card instead, avoiding the problem of someone corrupting a soldered down flash. – Chris Stratton Aug 01 '12 at 03:47
  • That is interesting. So is it the FAT partition that contains this booting data? or is it stored somewhere else on a hidden part of the SD? In theroy you could then in an even easier attempt, try to redirect the loading process to another "hacked" on device... – Piotr Kula Aug 01 '12 at 07:16
  • Here is the boot order http://raspberrypi.stackexchange.com/questions/1200/what-happens-during-the-boot-process It has been stated several times by RPi Foundation members that the boot order cannot be modified. The only option for a boot loader like feature would be to create a custom mini kernel.img to be your boot loader which would then call the actual kernel.img of your choice. – Dan B Aug 01 '12 at 13:25
  • In actuality the boot rom likely has other modes accessable via option pin strapping. It would be a very unusual SOC if it did not, and I doubt the pi project initially assumed the kinds of volumes needed to customize the mask. – Chris Stratton Aug 01 '12 at 14:41
0

USB HDD or USB SSD should give you the fastest performance. The SD cards write around 12 MB/s for class10 devices and USB can get up to 30 MB/s.

I do not agree that the USB hub is bottlenecking the ethernet on RPi, because the HUB can sustain 400 Mbit/s and the ethernet is 100. So theoretically it can sustain up to 4 x Fast ethernet 100Mb/s ports.

Angel Genchev
  • 1
  • EVERY class 10 SD I've ever tested is > 19mb/s write. I had a class 6 that did 14mb/s. SanDisk Extreme SD's that claim 45mb/s achieve 50mb/s read and 47mb/s write, which was a surprise. Lexar 400x SD's are the fastest I've tested at 70mb/s read, but about 30mb/s write. On USB 2.0, there's virtually no difference between fast SD cards, HDDs and SSDs. All the USB 2.0 interfaces I've messed with cap out at about 35mb/s. I've seen USB 2.0 thumb drives that cap out around 10mb/s and some even lower than that. Many USB 3.0 thumb drives have write speeds capped at 10-20 mb/s, even on USB 3.0 ports! – Therealstubot Aug 21 '14 at 00:48
0

I get max 4Mb/s using a USB flash drive and the same for USD HDD. This while simply dropping the files through the windows network 'homegroup' to the device connected to the Raspberry.

I was expecting at least 10 times that, but I haven't managed to get the transfer speed up. I tried FTP transfer directly to a USB HDD (this device transfers at +60Mb/s when connected to my laptop) but the results are very poor. 100KiB/s! Resulting in hours of traffic for only a few 100 MBs.

I now move the HDD between laptop and Raspberry Pi for data transfer :-(

goobering
  • 10,610
  • 4
  • 38
  • 64
NookaV
  • 1
  • I think you have a configuration/network problem because I can upload a file directly to my USB stick up to 24 Mbps. And I think I could do better with Ethernet instead of Wi-Fi. I have been using `vsftpd`. – Morgan Courbet Jan 04 '13 at 20:54