Build Apache Spark on Windows with Cygwin

Pull down the source.

git clone git://github.com/apache/spark.git

The build fails to generate this file:-

core\target\extra-resources\spark-version-info.properties

A working solution is to manually generate it and take it out of the build.

Convert the following script to UNIX line endings.

build\spark-build-info

Then run it with the parameters from the pom.

spark-build-info /cygdrive/c/work/spark/core/target/extra-resources 2.0.0-SNAPSHOT

Use dos2unix or Notepad++, any facility that can do it.

Run the maven-antrun-plugin command in the pom manually.

Comment out that plugin.

<!– run manually

<plugin>

<groupId>org.apache.maven.plugins</groupId>

<artifactId>maven-antrun-plugin</artifactId>

<executions>

<execution>

<phase>generate-resources</phase>

<configuration>

Execute the shell script to generate the spark build information.

<tasks>

<exec executable=”C:\cygwin64\bin\bash.exe C:/work/spark/build/spark-build-info”>

<arg value=”/cygdrive/c/work/spark/core/target/extra-resources”/>

<arg value=”${pom.version}”/>

</exec>

</tasks>

</configuration>

<goals>

<goal>run</goal>

</goals>

</execution>

</executions>

</plugin>

–>

 

Build while skipping tests.

mvn install -U -DskipTests=true