Elasticsearch Python Example

ElasticSearch

In this example notebook we show how to write/read data to/from Elasticsearch using spark. We use the dataset from American Kennel Club dog breed data

Setup

  1. Dowload the dataset

    wget -O akc_breed_info.csv https://query.data.world/s/msmjhcmdjslsvjzcaqmtreu52gkuno
    
  2. Upload akc_breed_info.csv to your resources dataset

Writing to Elasticsearch

from hops import elasticsearch, hdfs 

df = spark.read.option("header","true").csv("hdfs:///Projects/" + hdfs.project_name() + "/Resources/akc_breed_info.csv")
df.write.format(
    'org.elasticsearch.spark.sql'
).options(
    **elasticsearch.get_elasticsearch_config("newindex")
).mode("Overwrite").save()

Reading from Elasticsearch

reader = spark.read.format(
    "org.elasticsearch.spark.sql"
).options(**elasticsearch.get_elasticsearch_config("myindex"))
df = reader.load().orderBy("breed")
df.show()
+--------------------+------------------+-----------------+---------------+--------------+
|               Breed|height_high_inches|height_low_inches|weight_high_lbs|weight_low_lbs|
+--------------------+------------------+-----------------+---------------+--------------+
|       Affenpinscher|                12|                9|             12|             8|
|        Afghan Hound|                27|               25|             60|            50|
|     Airdale Terrier|                24|               22|             45|            45|
|               Akita|                28|               26|            120|            80|
|    Alaskan Malamute|                na|               na|             na|            na|
|     American Eskimo|                19|                9|             30|            25|
|   American Foxhound|                25|               22|             70|            65|
|American Stafford...|                19|               17|             50|            40|
|American Water Sp...|                18|               15|             45|            25|
|  Anatolian Sheepdog|                29|               27|            150|           100|
|Australian Cattle...|                20|               17|             45|            35|
| Australian Shepherd|                23|               18|             60|            40|
|  Australian Terrier|                10|               10|             14|            10|
|             Basenji|                17|               17|             22|            20|
|        Basset Hound|                14|               14|             50|            40|
|              Beagle|                16|               13|             30|            18|
|      Bearded Collie|                22|               20|             60|            40|
|           Beauceron|                27|               24|            120|           100|
|  Bedlington Terrier|                16|               15|             23|            18|
|    Belgian Malinois|                26|               22|             65|            60|
+--------------------+------------------+-----------------+---------------+--------------+
only showing top 20 rows