[time-nuts] How best to exchange Large files?
hmurray at megapathdsl.net
Wed Feb 22 00:43:56 UTC 2012
albertson.chris at gmail.com said:
> What a complex setup! Can't you user some of the Linux server's disk space
> and write the files directly to that disk? Seems that would be much
[snip of neat brick story]
> I've used this story in data processing system designs when I see people
> moving data around. I say "think, where is the final place you want your
> data. Then just put it there to begin with and be done with it."
> So, just open some kind of connection and put the data where it goes. Even a
> little Arduino has more then enough power for that.
The cost of that approach is another level of complexity at the system
administration level. You have to keep both systems running in order to
If I'm collecting interesting data, I prefer to keep the system simple. I
have one running on a UPS. I get a lot longer holdover by collecting data on
a stand alone small box than I would if the UPS had to run the "final" system
I set things up to break log files into day sized chunks. They get names
like foo.20120221, with foo being a link to the current one. (Scheme copied
from ntpd.) Most of the time, they are a reasonable size. Then I use
standard unix utilities to copy them over to where I process them. I think
rsync does most of the work. It's buried in a script.
One advantage of an explicit copy step is you get a layer of redundancy.
That may not be worth anything if you already have a good backup plan.
These are my opinions, not necessarily my employer's. I hate spam.
More information about the time-nuts