functions

Value Members

1. final def !=(arg0: AnyRef): Boolean

   Definition Classes

   AnyRef

2. final def !=(arg0: Any): Boolean

   Definition Classes

   Any

3. final def ##(): Int

   Definition Classes

   AnyRef → Any

4. final def ==(arg0: AnyRef): Boolean

   Definition Classes

   AnyRef

5. final def ==(arg0: Any): Boolean

   Definition Classes

   Any

6. def abs(e: Column): Column

   Computes the absolutle value.

7. def approxCountDistinct(columnName: String, rsd: Double): Column

   Aggregate function: returns the approximate number of distinct items in a group.

8. def approxCountDistinct(e: Column, rsd: Double): Column

   Aggregate function: returns the approximate number of distinct items in a group.

9. def approxCountDistinct(columnName: String): Column

   Aggregate function: returns the approximate number of distinct items in a group.

10. def approxCountDistinct(e: Column): Column

    Aggregate function: returns the approximate number of distinct items in a group.

11. final def asInstanceOf[T0]: T0

    Definition Classes

    Any

12. def asc(columnName: String): Column

    Returns a sort expression based on ascending order of the column.

    Returns a sort expression based on ascending order of the column. {{ // Sort by dept in ascending order, and then age in descending order. df.sort(asc("dept"), desc("age")) }}

13. def avg(columnName: String): Column

    Aggregate function: returns the average of the values in a group.

14. def avg(e: Column): Column

    Aggregate function: returns the average of the values in a group.

15. def callUDF(f: Function10[_, _, _, _, _, _, _, _, _, _, _], returnType: DataType, arg1: Column, arg2: Column, arg3: Column, arg4: Column, arg5: Column, arg6: Column, arg7: Column, arg8: Column, arg9: Column, arg10: Column): Column

    Call a Scala function of 10 arguments as user-defined function (UDF).

    Call a Scala function of 10 arguments as user-defined function (UDF). This requires you to specify the return data type.

16. def callUDF(f: Function9[_, _, _, _, _, _, _, _, _, _], returnType: DataType, arg1: Column, arg2: Column, arg3: Column, arg4: Column, arg5: Column, arg6: Column, arg7: Column, arg8: Column, arg9: Column): Column

    Call a Scala function of 9 arguments as user-defined function (UDF).

    Call a Scala function of 9 arguments as user-defined function (UDF). This requires you to specify the return data type.

17. def callUDF(f: Function8[_, _, _, _, _, _, _, _, _], returnType: DataType, arg1: Column, arg2: Column, arg3: Column, arg4: Column, arg5: Column, arg6: Column, arg7: Column, arg8: Column): Column

    Call a Scala function of 8 arguments as user-defined function (UDF).

    Call a Scala function of 8 arguments as user-defined function (UDF). This requires you to specify the return data type.

18. def callUDF(f: Function7[_, _, _, _, _, _, _, _], returnType: DataType, arg1: Column, arg2: Column, arg3: Column, arg4: Column, arg5: Column, arg6: Column, arg7: Column): Column

    Call a Scala function of 7 arguments as user-defined function (UDF).

    Call a Scala function of 7 arguments as user-defined function (UDF). This requires you to specify the return data type.

19. def callUDF(f: Function6[_, _, _, _, _, _, _], returnType: DataType, arg1: Column, arg2: Column, arg3: Column, arg4: Column, arg5: Column, arg6: Column): Column

    Call a Scala function of 6 arguments as user-defined function (UDF).

    Call a Scala function of 6 arguments as user-defined function (UDF). This requires you to specify the return data type.

20. def callUDF(f: Function5[_, _, _, _, _, _], returnType: DataType, arg1: Column, arg2: Column, arg3: Column, arg4: Column, arg5: Column): Column

    Call a Scala function of 5 arguments as user-defined function (UDF).

    Call a Scala function of 5 arguments as user-defined function (UDF). This requires you to specify the return data type.

21. def callUDF(f: Function4[_, _, _, _, _], returnType: DataType, arg1: Column, arg2: Column, arg3: Column, arg4: Column): Column

    Call a Scala function of 4 arguments as user-defined function (UDF).

    Call a Scala function of 4 arguments as user-defined function (UDF). This requires you to specify the return data type.

22. def callUDF(f: Function3[_, _, _, _], returnType: DataType, arg1: Column, arg2: Column, arg3: Column): Column

    Call a Scala function of 3 arguments as user-defined function (UDF).

    Call a Scala function of 3 arguments as user-defined function (UDF). This requires you to specify the return data type.

23. def callUDF(f: Function2[_, _, _], returnType: DataType, arg1: Column, arg2: Column): Column

    Call a Scala function of 2 arguments as user-defined function (UDF).

    Call a Scala function of 2 arguments as user-defined function (UDF). This requires you to specify the return data type.

24. def callUDF(f: Function1[_, _], returnType: DataType, arg1: Column): Column

    Call a Scala function of 1 arguments as user-defined function (UDF).

    Call a Scala function of 1 arguments as user-defined function (UDF). This requires you to specify the return data type.

25. def callUDF(f: Function0[_], returnType: DataType): Column

    Call a Scala function of 0 arguments as user-defined function (UDF).

    Call a Scala function of 0 arguments as user-defined function (UDF). This requires you to specify the return data type.

26. def clone(): AnyRef

    Attributes

    protected[java.lang]

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

27. def coalesce(e: Column*): Column

    Returns the first column that is not null.

    Returns the first column that is not null.

    df.select(coalesce(df("a"), df("b")))

    Annotations

    @varargs()

28. def col(colName: String): Column

    Returns a Column based on the given column name.

29. def column(colName: String): Column

    Returns a Column based on the given column name.

    Returns a Column based on the given column name. Alias of col.

30. def count(columnName: String): Column

    Aggregate function: returns the number of items in a group.

31. def count(e: Column): Column

    Aggregate function: returns the number of items in a group.

32. def countDistinct(columnName: String, columnNames: String*): Column

    Aggregate function: returns the number of distinct items in a group.

    Aggregate function: returns the number of distinct items in a group.

    Annotations

    @varargs()

33. def countDistinct(expr: Column, exprs: Column*): Column

    Aggregate function: returns the number of distinct items in a group.

    Aggregate function: returns the number of distinct items in a group.

    Annotations

    @varargs()

34. def desc(columnName: String): Column

    Returns a sort expression based on the descending order of the column.

    Returns a sort expression based on the descending order of the column. {{ // Sort by dept in ascending order, and then age in descending order. df.sort(asc("dept"), desc("age")) }}

35. final def eq(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef

36. def equals(arg0: Any): Boolean

    Definition Classes

    AnyRef → Any

37. def finalize(): Unit

    Attributes

    protected[java.lang]

    Definition Classes

    AnyRef

    Annotations

    @throws( classOf[java.lang.Throwable] )

38. def first(columnName: String): Column

    Aggregate function: returns the first value of a column in a group.

39. def first(e: Column): Column

    Aggregate function: returns the first value in a group.

40. final def getClass(): Class[_]

    Definition Classes

    AnyRef → Any

41. def hashCode(): Int

    Definition Classes

    AnyRef → Any

42. final def isInstanceOf[T0]: Boolean

    Definition Classes

    Any

43. def last(columnName: String): Column

    Aggregate function: returns the last value of the column in a group.

44. def last(e: Column): Column

    Aggregate function: returns the last value in a group.

45. def lit(literal: Any): Column

    Creates a Column of literal value.

    Creates a Column of literal value.

    The passed in object is returned directly if it is already a Column. If the object is a Scala Symbol, it is converted into a Column also. Otherwise, a new Column is created to represent the literal value.

46. def lower(e: Column): Column

    Converts a string exprsesion to lower case.

47. def max(columnName: String): Column

    Aggregate function: returns the maximum value of the column in a group.

48. def max(e: Column): Column

    Aggregate function: returns the maximum value of the expression in a group.

49. def min(columnName: String): Column

    Aggregate function: returns the minimum value of the column in a group.

50. def min(e: Column): Column

    Aggregate function: returns the minimum value of the expression in a group.

51. final def ne(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef

52. def negate(e: Column): Column

    Unary minus, i.

    Unary minus, i.e. negate the expression.

    // Select the amount column and negates all values.
    // Scala:
    df.select( -df("amount") )
    
    // Java:
    df.select( negate(df.col("amount")) );

53. def not(e: Column): Column

    Inversion of boolean expression, i.

    Inversion of boolean expression, i.e. NOT. {{ // Scala: select rows that are not active (isActive === false) df.filter( !df("isActive") )

    // Java: df.filter( not(df.col("isActive")) ); }}

54. final def notify(): Unit

    Definition Classes

    AnyRef

55. final def notifyAll(): Unit

    Definition Classes

    AnyRef

56. def sqrt(e: Column): Column

    Computes the square root of the specified float value.

57. def sum(columnName: String): Column

    Aggregate function: returns the sum of all values in the given column.

58. def sum(e: Column): Column

    Aggregate function: returns the sum of all values in the expression.

59. def sumDistinct(columnName: String): Column

    Aggregate function: returns the sum of distinct values in the expression.

60. def sumDistinct(e: Column): Column

    Aggregate function: returns the sum of distinct values in the expression.

61. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes

    AnyRef

62. def toString(): String

    Definition Classes

    AnyRef → Any

63. def udf[RT, A1, A2, A3, A4, A5, A6, A7, A8, A9, A10](f: (A1, A2, A3, A4, A5, A6, A7, A8, A9, A10) ⇒ RT)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[RT], arg1: scala.reflect.api.JavaUniverse.TypeTag[A1], arg2: scala.reflect.api.JavaUniverse.TypeTag[A2], arg3: scala.reflect.api.JavaUniverse.TypeTag[A3], arg4: scala.reflect.api.JavaUniverse.TypeTag[A4], arg5: scala.reflect.api.JavaUniverse.TypeTag[A5], arg6: scala.reflect.api.JavaUniverse.TypeTag[A6], arg7: scala.reflect.api.JavaUniverse.TypeTag[A7], arg8: scala.reflect.api.JavaUniverse.TypeTag[A8], arg9: scala.reflect.api.JavaUniverse.TypeTag[A9], arg10: scala.reflect.api.JavaUniverse.TypeTag[A10]): UserDefinedFunction

    Defines a user-defined function of 10 arguments as user-defined function (UDF).

    Defines a user-defined function of 10 arguments as user-defined function (UDF). The data types are automatically inferred based on the function's signature.

64. def udf[RT, A1, A2, A3, A4, A5, A6, A7, A8, A9](f: (A1, A2, A3, A4, A5, A6, A7, A8, A9) ⇒ RT)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[RT], arg1: scala.reflect.api.JavaUniverse.TypeTag[A1], arg2: scala.reflect.api.JavaUniverse.TypeTag[A2], arg3: scala.reflect.api.JavaUniverse.TypeTag[A3], arg4: scala.reflect.api.JavaUniverse.TypeTag[A4], arg5: scala.reflect.api.JavaUniverse.TypeTag[A5], arg6: scala.reflect.api.JavaUniverse.TypeTag[A6], arg7: scala.reflect.api.JavaUniverse.TypeTag[A7], arg8: scala.reflect.api.JavaUniverse.TypeTag[A8], arg9: scala.reflect.api.JavaUniverse.TypeTag[A9]): UserDefinedFunction

    Defines a user-defined function of 9 arguments as user-defined function (UDF).

    Defines a user-defined function of 9 arguments as user-defined function (UDF). The data types are automatically inferred based on the function's signature.

65. def udf[RT, A1, A2, A3, A4, A5, A6, A7, A8](f: (A1, A2, A3, A4, A5, A6, A7, A8) ⇒ RT)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[RT], arg1: scala.reflect.api.JavaUniverse.TypeTag[A1], arg2: scala.reflect.api.JavaUniverse.TypeTag[A2], arg3: scala.reflect.api.JavaUniverse.TypeTag[A3], arg4: scala.reflect.api.JavaUniverse.TypeTag[A4], arg5: scala.reflect.api.JavaUniverse.TypeTag[A5], arg6: scala.reflect.api.JavaUniverse.TypeTag[A6], arg7: scala.reflect.api.JavaUniverse.TypeTag[A7], arg8: scala.reflect.api.JavaUniverse.TypeTag[A8]): UserDefinedFunction

    Defines a user-defined function of 8 arguments as user-defined function (UDF).

    Defines a user-defined function of 8 arguments as user-defined function (UDF). The data types are automatically inferred based on the function's signature.

66. def udf[RT, A1, A2, A3, A4, A5, A6, A7](f: (A1, A2, A3, A4, A5, A6, A7) ⇒ RT)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[RT], arg1: scala.reflect.api.JavaUniverse.TypeTag[A1], arg2: scala.reflect.api.JavaUniverse.TypeTag[A2], arg3: scala.reflect.api.JavaUniverse.TypeTag[A3], arg4: scala.reflect.api.JavaUniverse.TypeTag[A4], arg5: scala.reflect.api.JavaUniverse.TypeTag[A5], arg6: scala.reflect.api.JavaUniverse.TypeTag[A6], arg7: scala.reflect.api.JavaUniverse.TypeTag[A7]): UserDefinedFunction

    Defines a user-defined function of 7 arguments as user-defined function (UDF).

    Defines a user-defined function of 7 arguments as user-defined function (UDF). The data types are automatically inferred based on the function's signature.

67. def udf[RT, A1, A2, A3, A4, A5, A6](f: (A1, A2, A3, A4, A5, A6) ⇒ RT)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[RT], arg1: scala.reflect.api.JavaUniverse.TypeTag[A1], arg2: scala.reflect.api.JavaUniverse.TypeTag[A2], arg3: scala.reflect.api.JavaUniverse.TypeTag[A3], arg4: scala.reflect.api.JavaUniverse.TypeTag[A4], arg5: scala.reflect.api.JavaUniverse.TypeTag[A5], arg6: scala.reflect.api.JavaUniverse.TypeTag[A6]): UserDefinedFunction

    Defines a user-defined function of 6 arguments as user-defined function (UDF).

    Defines a user-defined function of 6 arguments as user-defined function (UDF). The data types are automatically inferred based on the function's signature.

68. def udf[RT, A1, A2, A3, A4, A5](f: (A1, A2, A3, A4, A5) ⇒ RT)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[RT], arg1: scala.reflect.api.JavaUniverse.TypeTag[A1], arg2: scala.reflect.api.JavaUniverse.TypeTag[A2], arg3: scala.reflect.api.JavaUniverse.TypeTag[A3], arg4: scala.reflect.api.JavaUniverse.TypeTag[A4], arg5: scala.reflect.api.JavaUniverse.TypeTag[A5]): UserDefinedFunction

    Defines a user-defined function of 5 arguments as user-defined function (UDF).

    Defines a user-defined function of 5 arguments as user-defined function (UDF). The data types are automatically inferred based on the function's signature.

69. def udf[RT, A1, A2, A3, A4](f: (A1, A2, A3, A4) ⇒ RT)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[RT], arg1: scala.reflect.api.JavaUniverse.TypeTag[A1], arg2: scala.reflect.api.JavaUniverse.TypeTag[A2], arg3: scala.reflect.api.JavaUniverse.TypeTag[A3], arg4: scala.reflect.api.JavaUniverse.TypeTag[A4]): UserDefinedFunction

    Defines a user-defined function of 4 arguments as user-defined function (UDF).

    Defines a user-defined function of 4 arguments as user-defined function (UDF). The data types are automatically inferred based on the function's signature.

70. def udf[RT, A1, A2, A3](f: (A1, A2, A3) ⇒ RT)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[RT], arg1: scala.reflect.api.JavaUniverse.TypeTag[A1], arg2: scala.reflect.api.JavaUniverse.TypeTag[A2], arg3: scala.reflect.api.JavaUniverse.TypeTag[A3]): UserDefinedFunction

    Defines a user-defined function of 3 arguments as user-defined function (UDF).

    Defines a user-defined function of 3 arguments as user-defined function (UDF). The data types are automatically inferred based on the function's signature.

71. def udf[RT, A1, A2](f: (A1, A2) ⇒ RT)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[RT], arg1: scala.reflect.api.JavaUniverse.TypeTag[A1], arg2: scala.reflect.api.JavaUniverse.TypeTag[A2]): UserDefinedFunction

    Defines a user-defined function of 2 arguments as user-defined function (UDF).

    Defines a user-defined function of 2 arguments as user-defined function (UDF). The data types are automatically inferred based on the function's signature.

72. def udf[RT, A1](f: (A1) ⇒ RT)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[RT], arg1: scala.reflect.api.JavaUniverse.TypeTag[A1]): UserDefinedFunction

    Defines a user-defined function of 1 arguments as user-defined function (UDF).

    Defines a user-defined function of 1 arguments as user-defined function (UDF). The data types are automatically inferred based on the function's signature.

73. def udf[RT](f: () ⇒ RT)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[RT]): UserDefinedFunction

    Defines a user-defined function of 0 arguments as user-defined function (UDF).

    Defines a user-defined function of 0 arguments as user-defined function (UDF). The data types are automatically inferred based on the function's signature.

74. def upper(e: Column): Column

    Converts a string expression to upper case.

75. final def wait(): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

76. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

77. final def wait(arg0: Long): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )