caikit__caikit-233

The user is reporting a bug where inheriting from a `DataObjectBase` class and overriding methods without changing the schema (adding fields) causes a `TypeError: All DataObjectBase classes must follow dataclass syntax`.

This happens in `caikit/core/data_model/dataobject.py`, line 88, in `__new__`.

The provided sample code is:
```python
from caikit.core import DataObjectBase, dataobject

@dataobject
class Base(DataObjectBase):
    foo: int
    def doit(self):
        return self.foo * 2

@dataobject
class Derived(Base):
    def doit(self):
        return self.foo * 3
```

I need to investigate `caikit/core/data_model/dataobject.py` to see why it's raising this `TypeError`.

First, let's locate the relevant code and tests.

{
  "command": "grep -rn --include=\"*.py\" \"All DataObjectBase classes must follow dataclass syntax\" caikit/core/data_model/dataobject.py"
}

{
  "status": "ok",
  "stdout": "89:                    \"All DataObjectBase classes must follow dataclass syntax\"\n",
  "stderr": "",
  "exit_code": 0
}

Okay, the error message is indeed in `caikit/core/data_model/dataobject.py` around line 89.

Let's read the code around there.

{
  "end_line": 120,
  "filepath": "caikit/core/data_model/dataobject.py",
  "start_line": 50
}

{
  "status": "ok",
  "filepath": "caikit/core/data_model/dataobject.py",
  "content": "    JsonDict: struct_pb2.Struct,\n    np.int32: _descriptor.FieldDescriptor.TYPE_INT32,\n    np.int64: _descriptor.FieldDescriptor.TYPE_INT64,\n    np.uint32: _descriptor.FieldDescriptor.TYPE_UINT32,\n    np.uint64: _descriptor.FieldDescriptor.TYPE_UINT64,\n    np.float32: _descriptor.FieldDescriptor.TYPE_FLOAT,\n    np.float64: _descriptor.FieldDescriptor.TYPE_DOUBLE,\n    **PY_TO_PROTO_TYPES,\n}\n\n# Common package prefix\nCAIKIT_DATA_MODEL = \"caikit_data_model\"\n\n# Registry of auto-generated protos so that they can be rendered to .proto\n_AUTO_GEN_PROTO_CLASSES = []\n\n# Special attribute used to indicate which defaults are user provided\n_USER_DEFINED_DEFAULTS = \"__user_defined_defaults__\"\n\n## Public ######################################################################\n\n\nclass _DataObjectBaseMetaClass(_DataBaseMetaClass):\n    \"\"\"This metaclass is used for the DataObject base class so that all data\n    objects can delay the creation of their proto class until after the\n    metaclass has been instantiated.\n    \"\"\"\n\n    def __new__(mcs, name, bases, attrs):\n        \"\"\"When instantiating a new DataObject class, the proto class will not\n        yet have been generated, but the set of fields will be known since the\n        class will be the raw input representation of a @dataclass\n        \"\"\"\n\n        # Get the annotations that will go into the dataclass\n        if name != \"DataObjectBase\":\n            field_names = attrs.get(\"__annotations__\")\n            if field_names is None:\n                raise TypeError(\n                    \"All DataObjectBase classes must follow dataclass syntax\"\n                )\n            attrs[_DataBaseMetaClass._FWD_DECL_FIELDS] = field_names\n\n        # Delegate to the base metaclass\n        return super().__new__(mcs, name, bases, attrs)\n\n\nclass DataObjectBase(DataBase, metaclass=_DataObjectBaseMetaClass):\n    \"\"\"A DataObject is a data model class that is backed by a @dataclass.\n\n    Data model classes that use the @dataobject decorator must derive from this\n    base class.\n    \"\"\"\n\n\ndef dataobject(*args, **kwargs) -> Callable[[Type], Type[DataBase]]:\n    \"\"\"The @dataobject decorator can be used to define a Data Model object's\n    schema inline with the definition of the python class rather than needing to\n    bind to a pre-compiled protobufs class. For example:\n\n    @dataobject(\"foo.bar\")\n    @dataclass\n    class MyDataObject(DataObjectBase):\n        '''My Custom Data Object'''\n        foo: str\n        bar: int\n\n    NOTE: The wrapped class must NOT inherit directly from DataBase. That\n        inheritance will be added by this decorator, but if it is written\n        directly, the metaclass that links protobufs to the class will be called\n        before this decorator can auto-gen the protobufs class.",
  "start_line": 50,
  "end_line": 120,
  "total_lines": 386,
  "is_truncated": false
}

The `_DataObjectBaseMetaClass.__new__` method checks if `__annotations__` are present in `attrs`.

```python
    def __new__(mcs, name, bases, attrs):
        \"\"\"When instantiating a new DataObject class, the proto class will not\n        yet have been generated, but the set of fields will be known since the\n        class will be the raw input representation of a @dataclass\n        \"\"\"

        # Get the annotations that will go into the dataclass
        if name != \"DataObjectBase\":
            field_names = attrs.get(\"__annotations__\")
            if field_names is None:
                raise TypeError(
                    \"All DataObjectBase classes must follow dataclass syntax\"\n                )
            attrs[_DataBaseMetaClass._FWD_DECL_FIELDS] = field_names

        # Delegate to the base metaclass
        return super().__new__(mcs, name, bases, attrs)
```

In Python, if a class inherits from another and doesn't define any new annotations, `__annotations__` will not be in the `attrs` dictionary of the subclass. It will be in the `__annotations__` of the parent class.

The problem description says:
```python
@dataobject
class Base(DataObjectBase):
    foo: int
    def doit(self):
        return self.foo * 2

@dataobject
class Derived(Base):
    def doit(self):
        return self.foo * 3
```
In `Derived`, there are no annotations, so `attrs.get("__annotations__")` returns `None`, and the `TypeError` is raised.

To fix this, we should check if any of the base classes have annotations if the current class doesn't have any.

Wait, if it's a `DataObjectBase` class, it probably *should* have the fields of its parents.

Let's test this with a reproduction script.

{
  "command": "python3 - <<'PY'\nfrom caikit.core import DataObjectBase, dataobject\n\n@dataobject\nclass Base(DataObjectBase):\n    foo: int\n    def doit(self):\n        return self.foo * 2\n\ntry:\n    @dataobject\n    class Derived(Base):\n        def doit(self):\n            return self.foo * 3\n    print(\"Success!\")\nexcept TypeError as e:\n    print(f\"Caught expected TypeError: {e}\")\nexcept Exception as e:\n    print(f\"Caught unexpected exception: {type(e)}: {e}\")\nPY\n"
}

{
  "status": "ok",
  "stdout": "Caught expected TypeError: All DataObjectBase classes must follow dataclass syntax\n",
  "stderr": "<function register_backend_type at 0x7bca309ae170> is still in the BETA phase and subject to change!\n",
  "exit_code": 0
}

Confirmed. The reproduction script fails as expected.

Now, how to fix it. The `_DataObjectBaseMetaClass.__new__` should handle the case where the subclass doesn't define its own annotations but inherits from a `DataObjectBase` class that does.

If `field_names` is `None`, we should look for `__annotations__` in the base classes.

However, `DataObjectBase` itself might not have annotations. Only classes derived from it that actually define fields will.

Wait, if I have:
```python
@dataobject
class Base(DataObjectBase):
    foo: int

@dataobject
class Derived(Base):
    pass
```
`Base` will have `__annotations__ = {'foo': int}`.
`Derived` will not have `__annotations__` in its `attrs`.

If we want to support functional inheritance, we should allow subclasses to not define new fields.

If `attrs.get("__annotations__")` is `None`, we should check if any of the bases have `__annotations__`. But we should be careful not to just take any annotations; we probably want to inherit the fields.

Actually, `dataclasses` handles this by merging annotations.

In `_DataObjectBaseMetaClass.__new__`, `attrs[_DataBaseMetaClass._FWD_DECL_FIELDS] = field_names` is used to store the fields for later proto generation.

If we want `Derived` to have the same fields as `Base`, we should probably copy `Base`'s annotations if `Derived` has none.

But what if `Derived` *does* have annotations? Then it should probably be merged with `Base`'s annotations.

Let's see how `__annotations__` works in Python inheritance.
If `Base` has `__annotations__ = {'foo': int}` and `Derived(Base)` has `__annotations__ = {'bar': str}`, then `Derived.__annotations__` is `{'bar': str}`. It does NOT include `foo`.

To get all annotations, one usually has to traverse the MRO.

Wait, let's look at `_DataBaseMetaClass` to see what `_FWD_DECL_FIELDS` is used for.

{
  "end_line": 200,
  "filepath": "caikit/core/data_model/dataobject.py",
  "start_line": 120
}

{
  "status": "ok",
  "filepath": "caikit/core/data_model/dataobject.py",
  "content": "        before this decorator can auto-gen the protobufs class.\n\n    Args:\n        package:  str\n            The package name to use for the generated protobufs class\n\n    Returns:\n        decorator:  Callable[[Type], Type[DataBase]]\n            The decorator function that will wrap the given class\n    \"\"\"\n\n    def decorator(cls: Type) -> Type[DataBase]:\n        # Make sure that the wrapped class does NOT inherit from DataBase\n        error.value_check(\n            \"<COR95184230E>\",\n            issubclass(cls, (DataObjectBase, Enum)),\n            \"{} must inherit from DataObjectBase/Enum when using @dataobject\",\n            cls.__name__,\n        )\n\n        # Add the package to the kwargs\n        kwargs.setdefault(\"package\", package)\n\n        # If it's not an enum, fill in any missing field defaults as None\n        # and make sure it's a dataclass\n        if not issubclass(cls, Enum):\n            log.debug2(\"Wrapping data class %s\", cls)\n            user_defined_defaults = {}\n            for annotation in getattr(cls, \"__annotations__\", {}):\n                user_defined_default = getattr(cls, annotation, dataclasses.MISSING)\n                if user_defined_default == dataclasses.MISSING:\n                    log.debug3(\"Filling in None default for %s.%s\", cls, annotation)\n                    setattr(cls, annotation, None)\n                else:\n                    user_defined_defaults[annotation] = user_defined_default\n            # If the current __init__ is auto-generated by dataclass, remove\n            # it so that a new one is created with the new defaults\n            if _has_dataclass_init(cls):\n                log.debug3(\"Resetting default dataclass init\")\n                delattr(cls, \"__init__\")\n            cls = dataclasses.dataclass(cls)\n            setattr(cls, _USER_DEFINED_DEFAULTS, user_defined_defaults)\n\n        descriptor = _dataobject_to_proto(dataclass_=cls, **kwargs)\n\n        # Create the message class from the dataclass\n        proto_class = py_to_proto.descriptor_to_message_class(descriptor)\n        _AUTO_GEN_PROTO_CLASSES.append(proto_class)\n\n        # Add enums to the global enums module\n        for enum_class in _get_all_enums(proto_class):\n            log.debug2(\"Importing enum [%s]\", enum_class.DESCRIPTOR.name)\n            enums.import_enum(enum_class)\n\n        # Declare the merged class that binds DataBase to the wrapped class with\n        # this generated proto class\n        if isinstance(proto_class, type):\n            setattr(cls, \"_proto_class\", proto_class)\n            cls = _make_data_model_class(proto_class, cls)\n\n            # If this was a default-generated dataclass __init__ and there are\n            # any oneofs, we need to augment the __init__ to support kwargs for\n            # the individual fields\n            if _has_dataclass_init(cls) and cls._fields_oneofs_map:\n                setattr(cls, \"__init__\", _make_oneof_init(cls))\n\n        else:\n            enums.import_enum(proto_class, cls)\n            setattr(cls, \"_proto_enum\", proto_class)\n\n        # Return the decorated class\n        return cls\n\n    # If called without the function invocation, fill in the default argument\n    if args and callable(args[0]):\n        assert not kwargs, \"This shouldn't happen!\"\n        package = CAIKIT_DATA_MODEL\n        return decorator(args[0])\n\n    # Pull the package as an arg or a keyword arg\n    if args:",
  "start_line": 120,
  "end_line": 200,
  "total_lines": 386,
  "is_truncated": false
}

Wait, the `dataobject` decorator itself also does this:

```python
            user_defined_defaults = {}
            for annotation in getattr(cls, "__annotations__", {}):
                user_defined_default = getattr(cls, annotation, dataclasses.MISSING)
                # ...
            # ...
            cls = dataclasses.dataclass(cls)
```

If `cls` is `Derived(Base)` and `Derived` has no annotations, `getattr(cls, "__annotations__", {})` will be the annotations of `Base` because of Python's attribute lookup.

Wait, let's verify this.
In Python 3, if a class doesn't define `__annotations__`, it inherits it from its base class.

Example:
```python
class Base:
    x: int = 1

class Derived(Base):
    pass

print(Derived.__annotations__) # {'x': <class 'int'>}
```
YES, this is true.

So why does `_DataObjectBaseMetaClass.__new__` fail?
Because it looks in `attrs`. `attrs` is the dictionary of attributes defined *on the class being created*, not the final class object. At the time `__new__` is called, `Derived` has not been created yet, and `attrs` only contains things defined specifically in `Derived`'s body.

```python
    def __new__(mcs, name, bases, attrs):
        # ...
        # Get the annotations that will go into the dataclass
        if name != \"DataObjectBase\":
            field_names = attrs.get(\"__annotations__\")
            if field_names is None:
                raise TypeError(
                    \"All DataObjectBase classes must follow dataclass syntax\"\n                )
            attrs[_DataBaseMetaClass._FWD_DECL_FIELDS] = field_names
```

In the case of `Derived(Base)`, `attrs` does NOT contain `__annotations__`.

If we want to support this, we should check if any of the base classes have `__annotations__`.

But we should also consider what `_DataBaseMetaClass._FWD_DECL_FIELDS` is used for.

Let's look at `_DataBaseMetaClass`. It's not in `dataobject.py` (or is it?).

Wait, `class _DataObjectBaseMetaClass(_DataBaseMetaClass):`.
Let's find where `_DataBaseMetaClass` is defined.

{
  "command": "grep -rn \"_DataBaseMetaClass\" caikit/core/data_model/"
}

{
  "status": "ok",
  "stdout": "caikit/core/data_model/base.py:46:class _DataBaseMetaClass(type):\ncaikit/core/data_model/base.py:265:        _DataBaseMetaClass.class_registry[cls.full_name] = cls\ncaikit/core/data_model/base.py:363:                setattr(self, _DataBaseMetaClass._WHICH_ONEOF_ATTR, which_oneof)\ncaikit/core/data_model/base.py:413:class DataBase(metaclass=_DataBaseMetaClass):\ncaikit/core/data_model/base.py:482:        setattr(instance, _DataBaseMetaClass._BACKEND_ATTR, backend)\ncaikit/core/data_model/base.py:487:        return getattr(self, _DataBaseMetaClass._BACKEND_ATTR, None)\ncaikit/core/data_model/base.py:526:        which_oneof = getattr(self, _DataBaseMetaClass._WHICH_ONEOF_ATTR, None)\ncaikit/core/data_model/base.py:528:            super().__setattr__(_DataBaseMetaClass._WHICH_ONEOF_ATTR, {})\ncaikit/core/data_model/base.py:529:            which_oneof = getattr(self, _DataBaseMetaClass._WHICH_ONEOF_ATTR)\ncaikit/core/data_model/base.py:933:        cls = _DataBaseMetaClass.class_registry.get(proto_full_name)\ncaikit/core/data_model/base.py:968:        dm_class = _DataBaseMetaClass.class_registry.get(class_name)\ncaikit/core/data_model/base.py:973:            for full_name, dm_class in _DataBaseMetaClass.class_registry.items()\ncaikit/core/data_model/dataobject.py:40:from .base import DataBase, _DataBaseMetaClass\ncaikit/core/data_model/dataobject.py:72:class _DataObjectBaseMetaClass(_DataBaseMetaClass):\ncaikit/core/data_model/dataobject.py:91:            attrs[_DataBaseMetaClass._FWD_DECL_FIELDS] = field_names\ncaikit/core/data_model/dataobject.py:296:        _DataBaseMetaClass.parse_proto_descriptor(cls)\ncaikit/core/data_model/dataobject.py:307:                _DataBaseMetaClass.__new__(\ncaikit/core/data_model/dataobject.py:308:                    _DataBaseMetaClass,\ncaikit/core/data_model/dataobject.py:374:        setattr(self, _DataBaseMetaClass._WHICH_ONEOF_ATTR, which_oneof)\n",
  "stderr": "",
  "exit_code": 0
}

It's in `caikit/core/data_model/base.py`. Let's check what `_DataBaseMetaClass` does and what `_FWD_DECL_FIELDS` is.

{
  "filepath": "caikit/core/data_model/base.py"
}

{
  "status": "ok",
  "filepath": "caikit/core/data_model/base.py",
  "content": "# Copyright The Caikit Authors\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n\n\"\"\"Base classes and functionality for all data structures.\n\"\"\"\n\n# metaclass-generated field members cannot be detected by pylint\n# pylint: disable=no-member\n\n# Standard\nfrom dataclasses import dataclass\nfrom enum import Enum\nfrom typing import Any, Dict, Optional, Type, Union\nimport base64\nimport json\n\n# Third Party\nfrom google.protobuf import json_format\nfrom google.protobuf.descriptor import Descriptor\nfrom google.protobuf.internal import type_checkers as proto_type_checkers\nfrom google.protobuf.message import Message as ProtoMessageType\n\n# First Party\nimport alog\n\n# Local\nfrom ..toolkit.errors import error_handler\nfrom . import enums, json_dict\n\nlog = alog.use_channel(\"DATAM\")\nerror = error_handler.get(log)\n\n\nclass _DataBaseMetaClass(type):\n    \"\"\"Meta class for all structures in the data model.\"\"\"\n\n    # store a registry of all classes that use this metaclass, i.e.,\n    # all classes that extend DataBase. This is used for constructing new\n    # instances by name without having to introspect all modules in data_model.\n    class_registry = {}\n\n    # This sentinel value is used to determine whether a given attribute is\n    # present on a class without doing `getattr` twice in the case where the\n    # attribute does exist.\n    _MISSING_ATTRIBUTE = \"missing attribute\"\n\n    # Special attribute used to communicate that the proto fields are forward\n    # declared and will be populated after the metaclass has completed\n    # construction.\n    _FWD_DECL_FIELDS = \"__fwd_decl_fields__\"\n\n    # Special instance attributes that an instance of a class derived from\n    # DataBase may have. These are added to __slots__.\n    _BACKEND_ATTR = \"_backend\"\n    _WHICH_ONEOF_ATTR = \"_which_oneof\"\n\n    def __new__(mcs, name, bases, attrs):\n        \"\"\"When constructing a new data model class, we set the 'fields' class variable from the\n        protobufs descriptor and then set the '__slots__' magic class attribute to fields.  This\n        provides two benefits: (a) performance is improved since the classes only need to know\n        about these attributes (b) it helps to enforce that all member variables in these classes\n        are described in the protobufs.\n\n        Note:  If you want to add a variable for internal use that is not described in the\n        protobufs, it can be named in the tuple class variable _private_slots and will\n        automatically be added to __slots__.\n        \"\"\"\n\n        # Protobufs fields can be divided into these categories, which are used\n        # to automatically determine appropriate behavior in a number of methods\n        attrs[\"full_name\"] = name\n        attrs[\"fields_enum_map\"] = {}\n        attrs[\"fields_enum_rev\"] = {}\n        attrs[\"_fields_oneofs_map\"] = {}\n        attrs[\"_fields_to_oneof\"] = {}\n        attrs[\"_fields_map\"] = ()\n        attrs[\"_fields_message\"] = ()\n        attrs[\"_fields_message_repeated\"] = ()\n        attrs[\"_fields_enum\"] = ()\n        attrs[\"_fields_enum_repeated\"] = ()\n        attrs[\"_fields_primitive\"] = ()\n        attrs[\"_fields_primitive_repeated\"] = ()\n\n        # Look for the set of fields either from a predefined protobuf class or\n        # from a forward declaration from @dataobject\n        fields = ()\n        proto_class = None\n        if name not in [\"DataBase\", \"DataObjectBase\"]:\n            # Look for a precompiled proto class and if found, parse its\n            # descriptor\n            proto_class = attrs.get(\"_proto_class\")\n            if proto_class is not None:\n                all_oneof_fields = [\n                    field.name\n                    for oneof in proto_class.DESCRIPTOR.oneofs\n                    for field in oneof.fields\n                ]\n                fields = tuple(\n                    (\n                        field\n                        for field in proto_class.DESCRIPTOR.fields_by_name\n                        if field not in all_oneof_fields\n                    )\n                ) + tuple(proto_class.DESCRIPTOR.oneofs_by_name)\n\n            # Otherwise, we need to get the fields from a \"special\" attribute\n            else:\n                fields = attrs.pop(mcs._FWD_DECL_FIELDS, None)\n                log.debug4(\n                    \"Using dataclass forward declaration fields %s for %s\", fields, name\n                )\n                error.value_check(\n                    \"<COR49310991E>\",\n                    fields is not None,\n                    \"No proto class found for {}\",\n                    name,\n                )\n        attrs[\"fields\"] = fields\n        attrs[\"_proto_class\"] = proto_class\n\n        # Look if any private slots are declared as class variables\n        private_slots = attrs.setdefault(\"_private_slots\", ())\n\n        # Class slots are fields + private slots, this prevents other\n        # member attributes from being set and also improves performance\n        attrs[\"__slots__\"] = tuple(\n            [f\"_{field}\" for field in fields]\n            + list(private_slots)\n            + [mcs._BACKEND_ATTR, mcs._WHICH_ONEOF_ATTR]\n        )\n\n        # Create the instance of the type\n        instance = super().__new__(mcs, name, bases, attrs)\n\n        # If there's a valid proto class, perform proto descriptor parsing\n        if proto_class is not None:\n            mcs.parse_proto_descriptor(instance)\n",
  "start_line": 1,
  "end_line": 150,
  "total_lines": 990,
  "is_truncated": true
}

In `_DataBaseMetaClass.__new__`:

```python
            # Otherwise, we need to get the fields from a "special" attribute
            else:
                fields = attrs.pop(mcs._FWD_DECL_FIELDS, None)
                log.debug4(
                    "Using dataclass forward declaration fields %s for %s", fields, name
                )
                error.value_check(
                    "<COR49310991E>",
                    fields is not None,
                    "No proto class found for {}",
                    name,
                )
```

If `attrs.pop(mcs._FWD_DECL_FIELDS, None)` returns `None`, it raises an error `<COR49310991E>`.

Wait, the error message in the report is `TypeError: All DataObjectBase classes must follow dataclass syntax`. This comes from `_DataObjectBaseMetaClass.__new__`, not `_DataBaseMetaClass.__new__`.

```python
    def __new__(mcs, name, bases, attrs):
        # ...
        # Get the annotations that will go into the dataclass
        if name != \"DataObjectBase\":
            field_names = attrs.get(\"__annotations__\")
            if field_names is None:
                raise TypeError(
                    \"All DataObjectBase classes must follow dataclass syntax\"\n                )
            attrs[_DataBaseMetaClass._FWD_DECL_FIELDS] = field_names
        # Delegate to the base metaclass
        return super().__new__(mcs, name, bases, attrs)
```

If we change this to allow missing `__annotations__` in the subclass, `attrs[_DataBaseMetaClass._FWD_DECL_FIELDS]` will not be set. Then `_DataBaseMetaClass.__new__` will try to pop it and fail with `<COR49310991E>`.

So we need to ensure that `_FWD_DECL_FIELDS` is set even if the current class doesn't have `__annotations__`, as long as one of its base classes does.

If a subclass doesn't define its own annotations, it should probably inherit the fields from its parent.

How about this:
If `attrs.get("__annotations__")` is `None`, we check if any base class has `_FWD_DECL_FIELDS` (which was set by `_DataObjectBaseMetaClass` for the parent).

Wait, `_FWD_DECL_FIELDS` is set on the class being created, and then it's popped from `attrs` in `_DataBaseMetaClass.__new__` before the class is actually created.

Wait, let's look at `_DataBaseMetaClass.__new__` again.
`fields = attrs.pop(mcs._FWD_DECL_FIELDS, None)`

After `super().__new__(mcs, name, bases, attrs)`, the class is created. The `fields` are then stored in `attrs["fields"]`.

So once the parent class `Base` is created, it will have a `fields` attribute.

But `_DataObjectBaseMetaClass.__new__` is called *before* `_DataBaseMetaClass.__new__`.

If we want `Derived` to inherit fields from `Base`, we should find the `fields` of `Base` and use them if `Derived` has no annotations of its own.

But `Base`'s `fields` attribute is only set *after* `_DataBaseMetaClass.__new__` returns.

However, we can look at the `bases` tuple passed to `__new__`. The bases are already created classes.

So we can check if any of the `bases` have the `fields` attribute.

But the `fields` attribute in `_DataBaseMetaClass` is a tuple of protobuf field objects, whereas `_FWD_DECL_FIELDS` in `_DataObjectBaseMetaClass` is the `__annotations__` dictionary (which contains field names and types).

Actually, and this is important: if `Derived` has no annotations, it just wants to be a "functional" subclass, meaning it has the same schema as `Base`.

If it has the same schema, it should have the same `fields`.

Wait, if the user wants to add fields in `Derived`, they would add annotations.

If they don't add annotations, they just want the same fields as `Base`.

So, if `field_names = attrs.get("__annotations__")` is `None`, we should check if any of the `bases` have `fields`.

But `_DataBaseMetaClass.__new__` expects `_FWD_DECL_FIELDS` in `attrs` if no `_proto_class` is provided.

If we can't provide `_FWD_DECL_FIELDS` (the annotations), maybe we can provide `_proto_class`?

No, that's for precompiled protos.

Actually, for `DataObjectBase`, the `_proto_class` is generated from the dataclass.

If `Derived` is just a functional subclass of `Base`, it should use `Base`'s `_proto_class`.

Let's see how `_proto_class` is handled in `_DataBaseMetaClass.__new__`.

```python
        if name not in ["DataBase", "DataObjectBase"]:
            # Look for a precompiled proto class and if found, parse its
            # descriptor
            proto_class = attrs.get("_proto_class")
            if proto_class is not None:
                # ... (parse fields from proto_class)
            # Otherwise, we need to get the fields from a "special" attribute
            else:
                fields = attrs.pop(mcs._FWD_DECL_FIELDS, None)
                # ...
                error.value_check(
                    "<COR49310991E>",
                    fields is not None,
                    "No proto class found for {}",
                    name,
                )
```

If we set `attrs['_proto_class'] = Base._proto_class`, then it will work!

But `_DataObjectBaseMetaClass.__new__` is where the check for `__annotations__` happens.

Let's see:

```python
    def __new__(mcs, name, bases, attrs):
        # ...
        if name != "DataObjectBase":
            field_names = attrs.get("__annotations__")
            if field_names is None:
                # Potential fix here
                raise TypeError(
                    "All DataObjectBase classes must follow dataclass syntax"
                )
            attrs[_DataBaseMetaClass._FWD_DECL_FIELDS] = field_names

        return super().__new__(mcs, name, bases, attrs)
```

If `field_names` is `None`, we can check if any base class is a `DataObjectBase` (or `DataBase`) and has a `_proto_class`.

Wait, if `Base` was decorated with `@dataobject`, it will have `_proto_class`.

So we can do:
```python
        if name != "DataObjectBase":
            field_names = attrs.get("__annotations__")
            if field_names is None:
                # Check if we can inherit _proto_class from bases
                proto_class = getattr(bases[0], "_proto_class", None) if bases else None
                if proto_class is not None:
                    attrs["_proto_class"] = proto_class
                else:
                    raise TypeError(
                        "All DataObjectBase classes must follow dataclass syntax"
                    )
            else:
                attrs[_DataBaseMetaClass._FWD_DECL_FIELDS] = field_names
```

But what if there are multiple bases? (Rare for this system, but possible). We should probably check all bases.

And we should only do this if it's really supposed to be a `DataObjectBase`. The check `name != "DataObjectBase"` is already there.

Let's see if this makes sense. If `Derived` has no annotations, it's essentially a functional subclass. It will use the same protobuf schema as `Base`.

What about the `@dataobject` decorator?

```python
    def decorator(cls: Type) -> Type[DataBase]:
        # ...
        if not issubclass(cls, Enum):
            # ...
            for annotation in getattr(cls, "__annotations__", {}):
                # ...
            # ...
            cls = dataclasses.dataclass(cls)
```

If `Derived` has no annotations, `getattr(cls, "__annotations__", {})` will return `Base.__annotations__`.

Then `dataclasses.dataclass(cls)` will be called on `Derived`. Since `Derived` has no new fields, it will just have the fields of `Base`.

And the decorator then does:
```python
        descriptor = _dataobject_to_proto(dataclass_=cls, **kwargs)
        proto_class = py_to_proto.descriptor_to_message_class(descriptor)
```
Wait! This will create a NEW proto class for `Derived`, even if it has the same fields as `Base`.

Is that what we want? The problem says "Functional Inheritance".

"This will be very important for allowing content libraries to share common data structures with custom backend, construction, and functional interfaces"

If `Derived` has the same schema as `Base`, it should probably share the same `_proto_class` if possible, or at least have a compatible one.

But if `@dataobject` is used on `Derived`, the current implementation always creates a new `_proto_class`.

However, the `TypeError` we are talking about happens in the metaclass `__new__`, which is called *before* the `@dataobject` decorator is applied.

Wait, the decorator is applied *after* the class is defined.

```python
@dataobject
class Derived(Base):
    ...
```

The class `Derived` is created first. This is when `_DataObjectBaseMetaClass.__new__` is called.

If `_DataObjectBaseMetaClass.__new__` raises `TypeError`, the class is never created, and the decorator is never called.

So we MUST fix it in `_DataObjectBaseMetaClass.__new__`.

If we allow `field_names` to be `None` and we don't set `_FWD_DECL_FIELDS`, then `_DataBaseMetaClass.__new__` will fail unless we provide `_proto_class`.

Wait, if we provide `_proto_class` from `Base` to `Derived`, then `Derived` will have the same fields as `Base`. That sounds correct for a functional subclass.

But then the `@dataobject` decorator will run and create *another* `_proto_class` for `Derived` anyway?

Let's look at the decorator again.
```python
    def decorator(cls: Type) -> Type[DataBase]:
        # ...
        if not issubclass(cls, Enum):
            # ...
            cls = dataclasses.dataclass(cls)
        descriptor = _dataobject_to_proto(dataclass_=cls, **kwargs)
        proto_class = py_to_proto.descriptor_to_message_class(descriptor)
        # ...
        if isinstance(proto_class, type):
            setattr(cls, "_proto_class", proto_class)
            cls = _make_data_model_class(proto_class, cls)
```

Yes, it overrides `_proto_class`. So it doesn't matter if we set it in `__new__` as long as it satisfies `_DataBaseMetaClass.__new__`.

Actually, if we just remove the `TypeError` and don't set `_FWD_DECL_FIELDS`, then `_DataBaseMetaClass.__new__` will raise `<COR49310991E>`.

So we must set something that `_DataBaseMetaClass.__new__` accepts.

Either `_proto_class` or `_FWD_DECL_FIELDS`.

If we can't find animations in `attrs`, we can try to find them in base classes.

```python
            field_names = attrs.get("__annotations__")
            if field_names is None:
                # Try to find annotations in base classes
                for base in bases:
                    if hasattr(base, "__annotations__"):
                        field_names = base.__annotations__
                        break
                
                if field_names is None:
                    raise TypeError(
                        "All DataObjectBase classes must follow dataclass syntax"
                    )
            attrs[_DataBaseMetaClass._FWD_DECL_FIELDS] = field_names
```

But wait, if we do this, `field_names` will be the annotations of `Base`. Then `_DataBaseMetaClass._FWD_DECL_FIELDS` will be set to `Base.__annotations__`. Then `_DataBaseMetaClass.__new__` will use these to set `fields` on `Derived`.

This seems correct! `Derived` will have the same fields as `Base`.

Wait, one detail: `__annotations__` can be modified. We should probably copy them if we are passing them as `field_names`.

And also, `_DataBaseMetaClass._FWD_DECL_FIELDS` is popped in `_DataBaseMetaClass.__new__`.

Let's check if this works.

Reproduction again:
```python
from caikit.core import DataObjectBase, dataobject

@dataobject
class Base(DataObjectBase):
    foo: int
    def doit(self):
        return self.foo * 2

@dataobject
class Derived(Base):
    def doit(self):
        return self.foo * 3

b_inst = Base(1)
assert b_inst.doit() == 2

d_inst = Derived(1)
assert d_inst.doit() == 3
print("Success!")
```

If I apply the fix, this should pass.

Wait, what if `Base` also didn't have annotations? Then it would have raised `TypeError` when `Base` was being created. So at least one base class must have had annotations (or `Base` itself).

Actually, `DataObjectBase` doesn't have annotations. So `bases` will contain `Base` and `DataObjectBase`. `DataObjectBase` will not have `__annotations__`. `Base` will.

One more thing: what if the user defines `Derived` without `@dataobject`?
The `_DataObjectBaseMetaClass.__new__` is still called. The a la carte classes also need to satisfy the requirements of `_DataBaseMetaClass`.

The current code:
```python
    def __new__(mcs, name, bases, attrs):
        # ...
        if name != "DataObjectBase":
            field_names = attrs.get("__annotations__")
            if field_names is None:
                raise TypeError(
                    "All DataObjectBase classes must follow dataclass syntax"
                )
            attrs[_DataBaseMetaClass._FWD_DECL_FIELDS] = field_names

        return super().__new__(mcs, name, bases, attrs)
```

My proposed fix:
```python
    def __new__(mcs, name, bases, attrs):
        # ...
        if name != "DataObjectBase":
            field_names = attrs.get("__annotations__")
            if field_names is None:
                # Try to find annotations in base classes
                for base in bases:
                    # Use getattr to avoid issues with classes that don't have __annotations__
                    ann = getattr(base, "__annotations__", None)
                    if ann is not None:
                        field_names = ann
                        break
                
                if field_names is None:
                    raise TypeError(
                        "All DataObjectBase classes must follow dataclass syntax"
                    )
            attrs[_DataBaseMetaClass._FWD_DECL_FIELDS] = field_names

        return super().__new__(mcs, name, bases, attrs)
```

Wait, `getattr(base, "__annotations__", None)` will find `__annotations__` even if it's been inherited from further up. That's fine.

Is there any case where `field_names` being `None` should still raise `TypeError`?
If it doesn't inherit from any class that has annotations, then yes.

What if `Base` inherits from `DataObjectBase` and doesn't define any fields?
```python
@dataobject
class Base(DataObjectBase):
    pass
```
In this case, `Base` is created. `name != "DataObjectBase"`. `attrs.get("__annotations__")` is `None`. `bases` is `(DataObjectBase,)`. `DataObjectBase` has no `__annotations__`. So `field_names` remains `None`. `TypeError` is raised.

This is consistent with existing behavior. A `DataObjectBase` class MUST have a schema, either defined itself or inherited.

Let's double check if `DataObjectBase` has `__annotations__`.

{
  "command": "python3 -c \"from caikit.core import DataObjectBase; print(getattr(DataObjectBase, '__annotations__', 'No annotations'))\""
}

{
  "status": "ok",
  "stdout": "{}\n",
  "stderr": "<function register_backend_type at 0x7216d3e552d0> is still in the BETA phase and subject to change!\n",
  "exit_code": 0
}

Ah, `DataObjectBase` DOES have `__annotations__` and it is `{}`.

Wait, if `DataObjectBase` has `__annotations__ = {}`, then `getattr(DataObjectBase, "__annotations__", None)` will return `{}`.
Then `field_names` becomes `{}` (which is not `None`).
Then `attrs[_DataBaseMetaClass._FWD_DECL_FIELDS] = {}`.
Then `_DataBaseMetaClass.__new__` will set `fields = {}`.
And it will NOT raise `TypeError` or `<COR49310991E>`.

So if I use `if ann is not None: field_names = ann`, then `field_names` will be `{}`.

Is this okay?
If `Derived(Base)` is created, and `Base` has `foo: int`, then `Base.__annotations__` is `{'foo': int}`.
`getattr(Base, "__annotations__", None)` is `{'foo': int}`.
`field_names` is `{'foo': int}`. Correct.

If `Base(DataObjectBase)` is created, and it has no annotations.
`getattr(DataObjectBase, "__annotations__", None)` is `{}`.
`field_names` is `{}`. Correct.

Wait, if `Base(DataObjectBase)` has no annotations, should it fail?
The existing code:
```python
            field_names = attrs.get("__annotations__")
            if field_names is None:
                raise TypeError("All DataObjectBase classes must follow dataclass syntax")
```
If `Base(DataObjectBase)` has no annotations, `attrs.get("__annotations__")` is `None`. It raises `TypeError`.

So my proposed fix would CHANGE this behavior if I just use `getattr(base, "__annotations__", None)`.

Actually, I only want to succeed if I inherit from a class that *actually* defined some annotations, or if I'm okay with an empty schema.

But the issue is specifically about "Functional Inheritance" when the subclass doesn't add new fields.

If the base class already has fields, we want to inherit them.

If the base class has NO fields, then inheriting from it is the same as having NO fields.

Wait, if `DataObjectBase` has `__annotations__ = {}`, then any class inheriting from it should be able to have no fields?
No, because the existing code explicitly checks `attrs.get("__annotations__")`.

So we should only consider base classes that actually HAVE annotations defined on THEM, not just inherited.

To check if a class has annotations defined on it, we should look at `base.__dict__.get('__annotations__', None)`.

Let's check `DataObjectBase.__dict__`.

{
  "command": "python3 -c \"from caikit.core import DataObjectBase; print('__annotations__' in DataObjectBase.__dict__)\""
}

{
  "status": "ok",
  "stdout": "False\n",
  "stderr": "<function register_backend_type at 0x7a3d637352d0> is still in the BETA phase and subject to change!\n",
  "exit_code": 0
}

Wait, why did `getattr(DataObjectBase, "__annotations__", None)` return `{}` then?
Ah, it must be inheriting it from `DataBase` or some other base.

Let's check `DataBase`.

{
  "command": "python3 -c \"from caikit.core.data_model.base import DataBase; print('__annotations__' in DataBase.__dict__)\""
}

{
  "status": "ok",
  "stdout": "False\n",
  "stderr": "<function register_backend_type at 0x7989b7aed480> is still in the BETA phase and subject to change!\n",
  "exit_code": 0
}

That's strange. Where does `__annotations__` come from?
Maybe from `object`? No, `object` doesn't have `__annotations__`.

Let me check what `getattr(DataObjectBase, "__annotations__")` actually returns.

{
  "command": "python3 -c \"from caikit.core import DataObjectBase; print(getattr(DataObjectBase, '__annotations__'))\""
}

{
  "status": "ok",
  "stdout": "{}\n",
  "stderr": "<function register_backend_type at 0x7625dd2592d0> is still in the BETA phase and subject to change!\n",
  "exit_code": 0
}