Assert(가정 설정문) assert는 뒤의 조건이 True가 아니면 AssertError를 발생시킨다.
1 2 3 4 5 6 7 a = 3 assert a == 2 #결과 Traceback (most recent call last): File "<stdin>", line 1, in <module> AssertionError 왜 assert가 필요할까? 어떤 함수는 성능을 높이기 위해 반드시 정수만을 입력받아 처리하도록 만들 수 있다. 이런 함수를 만들기 위해서는 반드시 함수에 정수만 들어오는지 확인할 필요가 있다. 이를 위해 if문을 사용할 수도 있고 에외 처리를 사용할 수도 있지만 가정 설정문을 사용하는 방법도 있다.
Description /docs/source/development/testing.rst 파일을 읽으며 테스트를 시도하는 도중에 다음과 같은 문제가 발생했고, 이 문제를 수정하기로 생각했다.
1 2 3 4 24/12/16 20:44:42 WARN Utils: Service 'sparkDriver' could not bind on a random free port. You may check whether configuring an appropriate binding address. 24/12/16 20:44:42 ERROR SparkContext: Error initializing SparkContext. 에러에서 설명한 대로, PORT가 이미 사용중이라는 뜻으로 생각되어 PORT를 수정하니 에러가 해결되었다.
해결법에 대해서 docs에 수정해두었다.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 .
Kafka with Zookeeper Install 1 brew install kafka Homebrew’s default installation path will differ based on the chips: Macs with Apple Silicon will install kafka under /opt/homebrew/Cellar.
Binaries and scrips will be in /opt/homebrew/bin
Kafka configurations will be in /opt/homebrew/etc/kafka
Zookeeper configurations will be in /opt/homebrew/etc/zookeeper
The log.dirs config(the location of kafka data) will be set to /opt/homebrew/var/lib/kafka-logs
Setup the $PATH environment variable In order to easily access the kafka binaries, you can edit your PATH variable by adding the following line(edit the content to your system) to your system run commands(~/.
Configurations Docker Image We use official Airflow image. We have to install the necessary libraries and packages into the Airflow container. For that, we have to create a Dockerfile
1 2 3 4 5 6 FROM apache/airflow:2.10.2 USER airflow COPY requirements.txt /requirements.txt RUN pip install -r /requirements.txt 1 2 3 confluent-kafka cassandra-driver pymongo This Dockerfile will be used to install airflow:2.10.2. Then, it will install all necessary libraries in the requirements.
Chapter 9: Design a web crawler A web crawler is known as a robot or spider. It is widely used by search engines to discover new or updated content on the web. Content can be a web page, an image, a video, a PDF file, etc. A web crawler starts by collecting a few web pages and then follows links on those pages to collect new content.
Install Helm chart 1 brew install helm Install the Chart 1 2 3 4 5 6 7 8 9 {seilylook} 💎minikube start {seilylook} 💎helm repo add apache-airflow https://airflow.apache.org "apache-airflow" has been added to your repositories {seilylook} 💎 helm repo list NAME URL apache-airflow https://airflow.apache.org Upgrade the Chart 1 2 3 4 5 6 7 8 9 10 11 {seilylook} 💎 helm upgrade --install airflow apache-airflow/airflow --namespace airflow --create-namespace {seilylook} 💎 ~/Development/Devlog main ± kubectl get pods -n airflow -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES airflow-postgresql-0 1/1 Running 0 9m10s 10.
