Distributed Data Processing Environments
Bachelor in Data Science
2026/2027
The course aims at providing hands-on experience with state of the art distributed data processing environments. First, it addresses cloud computing concepts and tools. Then, it provides a bottom-up overview of distributed storage and processing technologies, emphasizing scalability and usability. Finally, it introduces basic systems and information security concepts, as required to safely use current distributed environments.
Learning outcomes
- Discuss key performance, scalability, and reliability trade-offs in distributed computing environments.
- Identify main cloud computing concepts and technologies.
- Employ cloud-based storage and computing resources for distributed data processing tasks.
- Contrast and select processing solutions for data science problems.
- Assess computer security and information security risks in cloud data storage and processing.
- Select and apply techniques to mitigate security risks.
Instructors
Grading
The grade has two components:
- Group project and discussion (50%).
- Written exam (50%), with at least 8/20.
Submitted projects must be fully authored by the students and must not contain materials (text, code, …) from third parties, obtained online, or using AI tools unless explicitly marked and authorized by the instructors. See Academic Regulations and Code of Ethical Conduct for more information.
Schedule
| # | Date | Topic | Mat. | Read |
|---|---|---|---|---|
| PL1 | 23/9/26 | Linux installation and basics. | A1 | |
| T1 | Introduction. | |||
| PL2 | 30/9/26 | Scripts and images. | ||
| T2 | Virtualization and cloud. | R1 1-5; R2 9,24 | ||
| PL3 | 7/10/26 | Infrastructure as code. | ||
| T3 | Cloud setup. | |||
| PL4 | 14/10/26 | Cloud access. | pdf zip | |
| T4 | Storage and files. | pdf pdf | R2 20; R3 | |
| PL5 | 21/10/26 | Storage and files. | pdf zip | |
| T5 | Query execution. | R4 4 | ||
| PL6 | 21/10/26 | Storage and files. | (cont.) | A |
| T6 | 7/11/25 | Query optimization. | 🗎 | R4 4 |
Bibliography
| # | Title |
|---|---|
| R1 | Fox, A., et al. Above the clouds: A Berkeley view of cloud computing Dept. Electrical Eng. and Comput. Sciences, University of California, Berkeley, Rep. UCB/EECS 28.13 (2009). |
| R2 | Nemeth, E., Snyder, G., Hein, T.R., Whaley, B., Mackin, D. UNIX and Linux System Administration Handbook (5th Edition), Addison-Wesley Professional, 2017. |
| R3 | Aditya Somani. A Data Engineer’s Guide to Columnar Storage |
| R4 | J. M. Hellerstein, M. Stonebraker, and J. Hamilton. Architecture of a Database System Foundations and Trends® in Databases, vol. 1, no. 2, pp. 141–259, 2007. |
| R5 | Jeffrey Dean and Sanjay Ghemawat. MapReduce: Simplified Data Processing on Large Clusters, Comm. ACM, 2008. |
| R6 | Dieter Gollmann. Computer Security. Wiley, 2011. |
| A1 | J. Pereira. Introdução ao Unix U. Minho, (2025). |
| A2 | Mark Needham, Michael Hunger and Michael Simons. DuckDB in Action Manning, 2024. |