v2: missing positions

[erizocosmico]

Positions seem to be a boolean true. At least in Python

[erizocosmico]

In Java I get correct positions. In Python there are no positions, so it doesn't seem to be a general issue

[bzz]

@erizocosmico I could not reproduce this so far - with the latest release RC2 and the test data using python driver v2.10.0

    val client = BblfshClient("localhost", 9432)
    val fileName = "src/test/resources/python_file.py"

    val fileContent = Source.fromFile(fileName).getLines.mkString("\n")    
    val resp = client.parse(fileName, fileContent)

    val node = resp.get
    println(s"Loaded Python:\n $node")    

Results in correct positions

(@type,JString(uast:Position)), (col,JUint(1)), (line,JUint(1)), (offset,JUint(0))

Let me know if that works for you, but in case it is not, if you could share a small reproducible example with the code and data I will be happy to take a deeper look!

[erizocosmico]

I'm gonna try to reproduce this again and give you some reproduction steps

[erizocosmico]

I've been trying a few things but I cannot reproduce that exact error. However, I can reproduce not reporting any positions in Python.

import org.bblfsh.client.v2.{BblfshClient, JNode}
import org.scalatest.{FlatSpec, Matchers}

class BblfshErrorCase extends FlatSpec with Matchers {
  "python" should "should have positions" in {
    val client = BblfshClient("localhost", 9432)
    val fileName = "src/test/resources/python_file.py"

    val fileContent =
      """#!/usr/bin/env python
        |
        |from __future__ import print_function
        |
        |if __name__ == '__main__':
        | print('hello world')
        |""".stripMargin
    val resp = client.parse(fileName, fileContent)

    val node = JNode.parseFrom(resp.uast.toByteArray)
    println(node)
  }
}

Results in:

JObject(ArrayBuffer((@pos,JObject(ArrayBuffer((@type,JString(uast:Positions))))), (@role,JArray(ArrayBuffer(JString(File), JString(Module)))), (@type,JString(python:Module)), (body,JArray(ArrayBuffer(JObject(ArrayBuffer((@pos,JObject(ArrayBuffer((@type,JString(uast:Positions)), (end,JObject(ArrayBuffer((@type,JString(uast:Position)), (col,JUint(16)), (line,JUint(3)), (offset,JUint(38))))), (start,JObject(ArrayBuffer((@type,JString(uast:Position)), (col,JUint(1)), (line,JUint(3)), (offset,JUint(23)))))))), (@type,JString(uast:RuntimeImport)), (All,JBool(false)), (Names,JArray(ArrayBuffer(JObject(ArrayBuffer((@pos,JObject(ArrayBuffer((@type,JString(uast:Positions))))), (@type,JString(uast:Identifier)), (Name,JString(print_function))))))), (Path,JObject(ArrayBuffer((@pos,JObject(ArrayBuffer((@type,JString(uast:Positions))))), (@type,JString(uast:Identifier)), (Name,JString(__future__))))), (Target,JNull()))), JObject(ArrayBuffer((@pos,JObject(ArrayBuffer((@type,JString(uast:Positions)), (end,JObject(ArrayBuffer((@type,JString(uast:Position)), (col,JUint(3)), (line,JUint(5)), (offset,JUint(64))))), (start,JObject(ArrayBuffer((@type,JString(uast:Position)), (col,JUint(1)), (line,JUint(5)), (offset,JUint(62)))))))), (@role,JArray(ArrayBuffer(JString(If), JString(Expression)))), (@token,JString(if)), (@type,JString(python:If)), (body,JObject(ArrayBuffer((@role,JArray(ArrayBuffer(JString(If), JString(Body), JString(Then)))), (@type,JString(python:If.body)), (body_stmts,JArray(ArrayBuffer(JObject(ArrayBuffer((@pos,JObject(ArrayBuffer((@type,JString(uast:Positions)), (start,JObject(ArrayBuffer((@type,JString(uast:Position)), (col,JUint(2)), (line,JUint(6)), (offset,JUint(90)))))))), (@role,JArray(ArrayBuffer(JString(Expression)))), (@type,JString(python:Expr)), (value,JObject(ArrayBuffer((@pos,JObject(ArrayBuffer((@type,JString(uast:Positions)), (start,JObject(ArrayBuffer((@type,JString(uast:Position)), (col,JUint(2)), (line,JUint(6)), (offset,JUint(90)))))))), (@role,JArray(ArrayBuffer(JString(Function), JString(Call), JString(Expression)))), (@type,JString(python:Call)), (args,JArray(ArrayBuffer(JObject(ArrayBuffer((@role,JArray(ArrayBuffer(JString(Function), JString(Call), JString(Positional), JString(Argument), JString(Name)))), (@type,JString(python:BoxedStr)), (boxed_value,JObject(ArrayBuffer((@pos,JObject(ArrayBuffer((@type,JString(uast:Positions)), (end,JObject(ArrayBuffer((@type,JString(uast:Position)), (col,JUint(21)), (line,JUint(6)), (offset,JUint(109))))), (start,JObject(ArrayBuffer((@type,JString(uast:Position)), (col,JUint(8)), (line,JUint(6)), (offset,JUint(96)))))))), (@type,JString(uast:String)), (Format,JString()), (Value,JString(hello world)))))))))), (func,JObject(ArrayBuffer((@role,JArray(ArrayBuffer(JString(Call), JString(Callee)))), (@type,JString(python:BoxedName)), (boxed_value,JObject(ArrayBuffer((@pos,JObject(ArrayBuffer((@type,JString(uast:Positions)), (end,JObject(ArrayBuffer((@type,JString(uast:Position)), (col,JUint(7)), (line,JUint(6)), (offset,JUint(95))))), (start,JObject(ArrayBuffer((@type,JString(uast:Position)), (col,JUint(2)), (line,JUint(6)), (offset,JUint(90)))))))), (@type,JString(uast:Identifier)), (Name,JString(print))))), (ctx,JString(Load))))), (keywords,JArray(ArrayBuffer()))))))))))))), (orelse,JObject(ArrayBuffer((@role,JArray(ArrayBuffer(JString(If), JString(Body), JString(Else)))), (@token,JString(else)), (@type,JString(python:If.orelse)), (else_stmts,JArray(ArrayBuffer()))))), (test,JObject(ArrayBuffer((@pos,JObject(ArrayBuffer((@type,JString(uast:Positions)), (start,JObject(ArrayBuffer((@type,JString(uast:Position)), (col,JUint(4)), (line,JUint(5)), (offset,JUint(65)))))))), (@role,JArray(ArrayBuffer(JString(Expression), JString(Binary), JString(Condition), JString(If)))), (@type,JString(python:Compare)), (comparators,JObject(ArrayBuffer((@role,JArray(ArrayBuffer(JString(Expression), JString(Right)))), (@type,JString(python:Compare.comparators)), (comparators,JArray(ArrayBuffer(JObject(ArrayBuffer((@type,JString(python:BoxedStr)), (boxed_value,JObject(ArrayBuffer((@pos,JObject(ArrayBuffer((@type,JString(uast:Positions)), (end,JObject(ArrayBuffer((@type,JString(uast:Position)), (col,JUint(26)), (line,JUint(5)), (offset,JUint(87))))), (start,JObject(ArrayBuffer((@type,JString(uast:Position)), (col,JUint(16)), (line,JUint(5)), (offset,JUint(77)))))))), (@type,JString(uast:String)), (Format,JString()), (Value,JString(__main__))))))))))))), (left,JObject(ArrayBuffer((@role,JArray(ArrayBuffer(JString(Expression), JString(Left)))), (@type,JString(python:BoxedName)), (boxed_value,JObject(ArrayBuffer((@pos,JObject(ArrayBuffer((@type,JString(uast:Positions)), (end,JObject(ArrayBuffer((@type,JString(uast:Position)), (col,JUint(12)), (line,JUint(5)), (offset,JUint(73))))), (start,JObject(ArrayBuffer((@type,JString(uast:Position)), (col,JUint(4)), (line,JUint(5)), (offset,JUint(65)))))))), (@type,JString(uast:Identifier)), (Name,JString(__name__))))), (ctx,JString(Load)), (noops_previous,JObject(ArrayBuffer((@pos,JObject(ArrayBuffer((@type,JString(uast:Positions)), (end,JObject(ArrayBuffer((@type,JString(uast:Position)), (col,JUint(1)), (line,JUint(4)), (offset,JUint(61))))), (start,JObject(ArrayBuffer((@type,JString(uast:Position)), (col,JUint(1)), (line,JUint(4)), (offset,JUint(61)))))))), (@role,JArray(ArrayBuffer(JString(Noop)))), (@type,JString(python:PreviousNoops)), (lines,JArray(ArrayBuffer())))))))), (ops,JObject(ArrayBuffer((@role,JArray(ArrayBuffer(JString(Expression)))), (@type,JString(python:Compare.ops)), (ops,JArray(ArrayBuffer(JObject(ArrayBuffer((@pos,JObject(ArrayBuffer((@type,JString(uast:Positions))))), (@role,JArray(ArrayBuffer(JString(Operator), JString(Relational), JString(Equal)))), (@token,JString(==)), (@type,JString(python:Eq))))))))))))))))))))

As you can see, the result contains no positions. Is there something I'm missing here?

Version is the latest RC2 snapshot and bblfsh server is v2.14.0-drivers.

[bzz]

Thanks for a detailed report! Will take a look and get back to you asap.

[bzz]

Here are several things I noticed:

1. Code snippet you post with rc2 is equivalent to:

   val resp = client.parse(fileName, fileContent)
   val node = resp.get

2. Positions are present in the output of the python driver using Scala client
   It's visible in the print output that you have shared e.g if I search for
   (@type,JString(uast:Position)), (col,JUint(16)) on this page.

3. There is a top-level node @pos that does not include any col/line/offset.
   But that is not specific for the scala-client, as that's the python-driver output, so it may be worth opening a new issue there. One way to verify that, that I find convenient is:

   bblfsh-cli -o json src/test/resources/python_file.py | jq . | less
   
   "@pos": {
     "@type": "uast:Positions"
   },
   ...
   

4. All this allowed me to find a new bug in a different case
   Serializing decoded UAST though a new context does not produce the same results.
   Going to be handled under Encoding UAST back does not produce the same bytes #104

Please let me know if that addresses your concerns or in case you have any other questions/feedback!