Scientific data management

Experiments and simulations now produce data faster than many research teams can comfortably handle, and the hours spent wrangling it are hours not spent on the science itself. This book examines current methods and tools for working with very large, varied scientific datasets so that investigators can stay focused on their questions.

Storage comes first. Early chapters explain how large volumes of data can be written and read without holding back running simulations or the analysis and visualization that depend on them. Later sections turn to moving data efficiently, managing storage space, and newer database systems designed for scientific work.

The book then covers how to arrange data for analysis, how to search very large collections effectively, how to automate multistep scientific workflows, and how to capture metadata and lineage information automatically. Together these chapters trace data from the moment it is generated through to its analysis.

Numerous detailed color images accompany the text, and the examples come from biology, ecology, geology, climatology, and other fields.