[Python] 대용량 데이터 처리 및 분석을 위한 PyArrow (Apache Arrow)

---

PyArrow란?

인메모리 분석을 위한 개발 플랫폼인 Apache Arrow의 Python용 라이브러리이다. Apache Arrow는 빅 데이터 시스템이 데이터를 빠르게 처리하고 이동할 수 있도록 하는 일련의 기술이 포함되어 있다. 최신 하드웨어에서 효율적인 분석 작업을 위해 구성된 플랫 및 계층적 데이터에 대해 표준화된 언어 독립적 열 메모리 형식을 지원한다.

 

 

호환성

PyArrow는 현재 Python 3.7, 3.8, 3.9 및 3.10과 호환됩니다. Windows, macOS 및 다양한 Linux 배포판(Ubuntu 16.04, Ubuntu 18.04 포함)에서 지원하고, 64비트 시스템을 권장합니다.

설치

pip install pyarrow

Windows에서 핍 휠 가져오기 문제가 발생하면 Visual Studio 2015용 Visual C++ 재배포 가능 패키지를 설치해야 할 수 있습니다 .

 

파일 읽기

URI로 파일 시스템을 유추하거나, filesystem 명시적으로 선언하여 pyarrow.parquet.read_table() 함수로 파일을 읽을 수 있다. 

from pyarrow import fs
import pyarrow.parquet as pq

# reading file.
# using a URI -> filesystem is inferred
pq.read_table("s3://my-bucket/data.parquet")
# using a path and filesystem
s3 = fs.S3FileSystem(..)
pq.read_table("my-bucket/data.parquet", filesystem=s3)

파일쓰기

FileSystem 인터페이스를 사용하면 파일을 읽거나 쓰기를 위해 파일을 열 수 있으면 파일류 객체처럼 코드를 구현할 수 있습니다. 

from pyarrow import fs
import pyarrow as pa

local = fs.LocalFileSystem()

with local.open_output_stream("test.arrow") as file:
   with pa.RecordBatchFileWriter(file, table.schema) as writer:
      writer.write_table(table)

---

728x90

---

NumPy 와의 호환

# NumPy to Arrow
import numpy as np
import pyarrow as pa
data = np.arange(10, dtype='int16')
arr1 = pa.array(data)
arr1

# Arrow to NumPy
arr2 = pa.array([4, 5, 6], type=pa.int32())
view = arr2.to_numpy()
view

 

<관련 포스팅>

2024.10.12 - [Python] 대용량 데이터 처리 및 분석을 위한 Duck DB

Pandas의 한계를 극복한 5가지 라이브러리: Dask, Vaex, Modin, Cudf, Polars

2023.09.21 - Pandas의 한계를 극복한 5가지 라이브러리: Dask, Vaex, Modin, Cudf, Polars

 

 

dataframe을 apply 연산을 멀티프로세스로 처리하는 방법을 소개합니다. 

dataframe에서 복잡하고, 오래 걸리는 연산이 필요한 경우 유용한 방법이 될 수 있습니다. 

[python] dataframe apply() multiprocessing

 

 

PyArrow Cookbook

• Reading and Writing Data
  • Write a Parquet file
  • Reading a Parquet file
  • Reading a subset of Parquet data
  • Saving Arrow Arrays to disk
  • Memory Mapping Arrow Arrays from disk
  • Writing CSV files
  • Writing CSV files incrementally
  • Reading CSV files
  • Writing Partitioned Datasets
  • Reading Partitioned data
  • Reading Partitioned Data from S3
  • Write a Feather file
  • Reading a Feather file
  • Reading Line Delimited JSON
  • Writing Compressed Data
  • Reading Compressed Data
• Creating Arrow Objects
  • Creating Arrays
  • Creating Tables
  • Create Table from Plain Types
  • Creating Record Batches
  • Store Categorical Data
• Working with Schema
  • Setting the data type of an Arrow Array
  • Setting the schema of a Table
  • Merging multiple schemas
• Data Manipulation
  • Computing Mean/Min/Max values of an array
  • Counting Occurrences of Elements
  • Applying arithmetic functions to arrays.
  • Appending tables to an existing table
  • Adding a column to an existing Table
  • Searching for values matching a predicate in Arrays
  • Filtering Arrays using a mask
• Arrow Flight
  • Simple Parquet storage service with Arrow Flight
  • Streaming Parquet Storage Service
  • Authentication with user/password

 

PyArrow Document

• Installing PyArrow
  • System Compatibility
  • Python Compatibility
  • Using Conda
  • Using Pip
  • Installing from source
  • Installing Nightly Packages
• Getting Started
  • Creating Arrays and Tables
  • Saving and Loading Tables
  • Performing Computations
  • Working with large data
  • Continuining from here
• Data Types and In-Memory Data Model
  • Type Metadata
  • Schemas
  • Arrays
  • Record Batches
  • Tables
  • Custom Schema and Field Metadata
• Compute Functions
  • Standard Compute Functions
  • Grouped Aggregations
• Memory and IO Interfaces
  • Referencing and Allocating Memory
  • Input and Output
• Streaming, Serialization, and IPC
  • Writing and Reading Streams
  • Efficiently Writing and Reading Arrow Data
  • Arbitrary Object Serialization
• Filesystem Interface
  • Usage
  • Local FS
  • S3
  • Hadoop Distributed File System (HDFS)
  • Using fsspec-compatible filesystems with Arrow
  • Using Arrow filesystems with fsspec
• Filesystem Interface (legacy)
  • Hadoop File System (HDFS)
• The Plasma In-Memory Object Store
  • The Plasma API
  • Using Arrow and Pandas with Plasma
  • Using Plasma with Huge Pages
• NumPy Integration
  • NumPy to Arrow
  • Arrow to NumPy
• Pandas Integration
  • DataFrames
  • Series
  • Handling pandas Indexes
  • Type differences
  • Nullable types
  • Memory Usage and Zero Copy
• Timestamps
  • Arrow/Pandas Timestamps
  • Timestamp Conversions
• Reading and Writing CSV files
  • Usage
  • Customized parsing
  • Customized conversion
  • Incremental reading
  • Character encoding
  • Customized writing
  • Incremental writing
  • Performance
• Feather File Format
  • Using Compression
  • Writing Version 1 (V1) Files
• Reading JSON files
  • Usage
  • Automatic Type Inference
  • Customized parsing
• Reading and Writing the Apache Parquet Format
  • Obtaining pyarrow with Parquet Support
  • Reading and Writing Single Files
  • Finer-grained Reading and Writing
  • Inspecting the Parquet File Metadata
  • Data Type Handling
  • Compression, Encoding, and File Compatibility
  • Partitioned Datasets (Multiple Files)
  • Writing to Partitioned Datasets
  • Reading from Partitioned Datasets
  • Using with Spark
  • Multithreaded Reads
  • Reading from cloud storage
• Tabular Datasets
  • Reading Datasets
  • Filtering data
  • Projecting columns
  • Reading partitioned data
  • Reading from cloud storage
  • Reading from Minio
  • Working with Parquet Datasets
  • Manual specification of the Dataset
  • Iterative (out of core or streaming) reads
  • A note on transactions & ACID guarantees
  • Writing Datasets
• Extending pyarrow
  • Controlling conversion to pyarrow.Array with the __arrow_array__ protocol
  • Defining extension types (“user-defined types”)
• PyArrow Integrations
  • Integrating PyArrow with R
  • Using pyarrow from C++ and Cython Code
  • CUDA Integration
• Getting Involved
• Benchmarks
  • Running the benchmarks
  • Running for arbitrary Git revisions
  • Compatibility

 

API Reference

• Data Types and Schemas
  • Factory Functions
  • Utility Functions
  • Type Classes
  • Type Checking
• Arrays and Scalars
  • Factory Functions
  • Array Types
  • Scalars
• Buffers and Memory
  • In-Memory Buffers
  • Memory Pools
• Compute Functions
  • Aggregations
  • Arithmetic Functions
  • Bit-wise Functions
  • Rounding Functions
  • Logarithmic Functions
  • Trigonometric Functions
  • Comparisons
  • Logical Functions
  • String Predicates
  • String Transforms
  • String Padding
  • String Trimming
  • String Splitting
  • String Component Extraction
  • String Joining
  • String Slicing
  • Containment Tests
  • Categorizations
  • Selecting / Multiplexing
  • Conversions
  • Temporal Component Extraction
  • Temporal Difference
  • Timezone Handling
  • Associative Transforms
  • Selections
  • Sorts and Partitions
  • Structural Transforms
  • Compute Options
• Streams and File Access
  • Factory Functions
  • Stream Classes
  • File Systems
• Tables and Tensors
  • Factory Functions
  • Classes
  • Tensors
• Serialization and IPC
  • Inter-Process Communication
  • Serialization
• Arrow Flight
  • Common Types
  • Flight Client
  • Flight Server
  • Authentication
  • Middleware
• Tabular File Formats
  • CSV Files
  • Feather Files
  • JSON Files
  • Parquet Files
  • ORC Files
• Filesystems
  • Interface
  • Filesystem Implementations
  • Utilities
• Dataset
  • Factory functions
  • Classes
• Plasma In-Memory Object Store
  • Classes
• CUDA Integration
  • CUDA Contexts
  • CUDA Buffers
  • Serialization and IPC
• Miscellaneous
  • Multi-Threading
  • Using with C extensions

---

<관련 링크>

PyArrow 공식 문서 : https://arrow.apache.org/docs/python/getstarted.html

Apache Arrow Github : https://github.com/apache/arrow

---