November 21, 2013 - by Christopher Hogue, Ph.D.
Ready to do queries (like MapReduce) for fast analysis of your existing server logs without ETL (extract/transform/load) steps?
Here is an update of ways to sync or move your server's existing log data to the Manta Storage Service.
For a fast introduction to Manta, Trevor O, co-founder of Caliper, wrote a great how-to blog. Trevor shows off Manta's Node.js based command line interface (CLI), which is a great place to start experimenting for moving files.
Voxer contributor Dave Eddy provides an open source Node.js utility called manta-sync for syncing local directories to remote Manta directories.
Joyent's own Ben Wen wrote a Powershell script for bulk moving files from a Windows directory hierarchy onto Manta which you can find here.
Wanelo sends its logs from 20 million events per day from over 40 hosts to Manta. Take a look at their blog post, slide deck and video here to see how they use MapReduce on Manta to better analyze customer behavior.
Don't to forget that in addition to Node.js, Joyent's Manta Software Development Kits (SDK) provide developers with customization options to move, organize and compute on data with Manta in Python, Ruby and Java.
Upload a few multi-Gig log files (or larger!) and see what you can learn with Manta!